Facial Recognition Software: An Update on Quickly Developing Tech Our previous post discussed the advances in facial recognition for travel and security applications. Another eagerly anticipated application of facial recognition technology is the rumored inclusion of this feature in an upcoming iPhone model and other contemporary phones. While Apple is notoriously tight-lipped about future products,...Read more
3D Conversion Patents take Center Stage in Hollywood Visual Effects Case
As everyone prepares for this summer’s blockbusters, Hollywood executives and patent professionals, alike, are getting their popcorn ready for a different show. Two of the biggest post-production/3D-conversion companies are preparing for battle in a patent infringement suit that is sure to create enemies and allies in the world of film post-production.
In recent years, it’s been almost a given for action-adventure theatrical releases to have 3D versions presented next-door to the traditional two-dimensioned film. Higher ticket prices for these unique theater experiences make 3D versions attractive to studios and distributors looking to bring in more cash for opening weekends. Also, 3D-capable high definition televisions and higher-capacity Blu-ray Discs have made the at-home 3D experience an extra revenue stream in a mediocre DVD sales/rental market. On the other hand, movie critics are always quick to suggest of marketplace fatigue for 3D films and some viewers complain of headaches from looking through the 3D glasses.
So, whether you are a fan of the productions or not, this pending patent lawsuit will bring to the foreground the fact that 3D movie technology is now a mainstream money-maker that affects the entire film industry.
Prime Focus Creative Services Canada filed a patent infringement suit against Legend3D in the U.S. District Court for the Central District of California. In the March 30, 2015 complaint, Prime Focus World requested a jury trial, an injunction banning Legend3D from performing the patented process, a finding of willful infringement and unspecified monetary damages.
Both companies primarily operate in the fields of visual effects and are known for their work in 3D conversion—i.e., taking a set of 2D images and preparing the images for viewing in a 3D venue.
Each company has some of the biggest blockbusters in recent memory. Prime Focus World lists Teenage Mutant Ninja Turtles, Edge of Tomorrow, Marvel’s Guardians of the Galaxy in its credits, as well as The Avengers: Age of Ultron to be released on May 1, 2015. Prime Focus also worked with some of the Star Wars prequels and the final Harry Potter films. They have won several awards from the International 3D and Advanced Imaging Society for their work on Sin City: A Dame to Kill For and Gravity.
Legend3D has just as impactful a résumé, working with mega films such as The Lego Movie, The Amazing Spider-Man 1 and 2, Man of Steel , Life of Pi, Jupiter Ascending and several Disney projects, including Maleficent, Oz the Great and Powerful, and Pirates of the Caribbean On Stranger Tides. Legend3D also worked on the Smurfs movies and the Shrek trilogy. This year’s Poltergeist reboot, in theaters May 22, is featured prominently on the Legend3D website.
Both PFW and Legend3D assert their respective contributions to Industrial Light and Magic and director Michael Bay for 2014’s Transformers: Age of Extinction. The most recent Transformers movie, and the start of a new trilogy, used a hybrid of natively captured 3D photography as well as converted shots. Legend3D boasts use as a conversion vendor that blended Bay’s 8 different camera formats, as well as multiple video effects (“vfx”) shots for the 2D and 3D versions of the film. PFW claims to have delivered a combination of more than 70 minutes of stereo effects and 2D visual effects from the film, citing work on adding depth to faces on long-range shots and augmenting the 3D appearance of smoke, snow and other particles.
The ’628 Patent and the Disputed Technology
The dispute revolves around U.S. Patent 8,922,628, issued on December 30, 2014 to inventor Chris Bond, assigned to Prime Focus Vfx Services, and titled “System and process for transforming two-dimensional images into three-dimensional images.” The patent was filed in 2010, and claims priority to a provisional patent filed September 1, 2009, which would indicate that the patent is estimated to expire in 2029.
The patent claims a post-production process that takes a 2D-filmed movie and makes it a 3D movie—specifically creating it frame by frame. Many misconceptions about 3D conversion (as opposed to natively filming it with a 3D camera rig) depict the process as a mindless, computer conversion or a factory-like process. However, 3D conversion is far more nuanced and requires artistic as well as geometric considerations when analyzing each frame and creating a three dimensional effect from a two dimensional image. Nevertheless, the ’628 Patent’s subject matter aims to simplify the estimated 4-6 month-long conversion process.
The background of the patent’s specification describes how a human’s eyes are essentially tricked into seeing three dimensions when each eye is respectively shown a two-dimensional image from a slightly different perspective. Such a “stereo image pair” has a left eye image and a right eye image, which the viewer’s visual system will naturally combine and create a perception of depth.
In the theater, stereo vision is typically created with two projectors and 3-D glasses. Each projector has a lens which polarizes the projector’s light in a different manner, so the left-eye viewing lens of the glasses screens out light from the right projector, and vice versa. Hence, the left eye only sees the “left eye image” and the right eye only sees the “right eye image,” and each viewer’s visual system combines the two and perceives depth in the image accordingly. When you take your glasses off, you see both. Not-so-long-ago, this was done with red- and blue-tinted glasses that ensured each eye was seeing the corresponding image. Today, this process can be recreated with a single projector or on a 3D-capable TV, which alternately displays each eye’s image at a high frame rate and accordingly blocks the eye-specific- image for the opposite eye via polarization. (NB: the polarized lenses of the glasses have to block some of the light out, so brightness is always a worry with 3D film presentation on screen and TV; look for big improvements in brightness and colors in the near future).
The ’628 Patent addresses issues in the conversion process and illustrates a method of analyzing a two-dimensional image to create a “working image” that will become the image from another perspective in a stereo image pair. In general, the selection and remapping process independently alters portions of the working image, so that “the remappings shift the image elements in a manner which produces a stereo depth effect” when the images are viewed together as a 3D image with the proper glasses or device.
The patent lists three key issues with present industry-accepted approaches for 3D conversion:
- manually outlining each element in a frame and extending that over a filmed sequence, one frame at a time (e., “rotoscoping”),
- virtually reconstructing the 3D geometry of the scene represented by the image, and
- filling in the “holes” that were previously occluded (e., background features that were blocked) prior to shifting 2D elements.
The patent proposes that by hiding certain image information, converters can achieve the same perceived stereoscopic effect as generating new information to fill those holes. This means that instead of moving objects in the second stereo image, reconstructing the 3D geometry and filling in the holes, each object could be warped or distorted to shrink/increase portions of the image to create a stereo counterpart. The patent specification purports that this method is quicker and results in a much more detailed, realistic final 3D image.
Specifically, independent claim 1 recites creating a “depth selection mask” in the working image by selecting certain features and assigning a depth selection value to each pixel location of the working image in the depth selection mask. Then, using a vector field of each pixel, the depth selection mask is applied to create a weighted displacement mask having a displacement vector associated with each pixel. Using the magnitude of the vector for each pixel in the working image, each pixel is displaced.
From the specification it is evident that each pixel of an object in an image is not meant to move an identical distance—that, depending on the respective vectors in the vector field, some image elements will be stretched, shrunk, or enlarged. As such, this warping and distorting is meant to create an image that appears to come from a different vantage point from the first, and thus create a stereo image pair.
Potential infringement of the independent claims may possibly be found from an expert’s close study of each of a movie-frame’s stereo image pair. If the proposed patent’s novelty really is the stretching of the image elements to create a second-eye image, over the alleged prior art’s shifting of features, analysis may reveal the process used for conversion from the product. For instance, displacement vector analysis, along with an optical study, could potentially reveal how the second image was created. Creating a correlation matrix for each image pair and overlaying displacement vectors on the images could potentially determine which features were moved/stretched and for what distance. However, like the 3D conversion process itself, analyzing each stereo pare frame-by-frame—of each converted film—could be excruciatingly long.
The direct defenses to infringement that Legend3D would most likely assert at this stage are (1) that they do not perform any/all the claimed steps or elements of the claims in the ’628 Patent and/or (2) that Prime Focus’s ’628 Patent is invalid because of (a) prior art rendering the claimed subject matter obvious at the time of invention or (b) alleging that the patent claims unpatentable subject matter, such as an abstract idea (i.e., the notorious 2014 Alice Supreme Court case).
The Alice-challenge (2b) is typically thrown in for every case these days, but may be omitted, as it could potentially put the validity of the defendant’s own patented technology in the spotlight. But it would certainly be a surprise if Legend3D did not utilize an Inter Partes Review at the U.S. Patent and Trademark Office to challenge the validity of the ’628 Patent in view of prior art (2a).
From publically available information, such as these 2012 case studies by Legend3D, it is unclear if anything like the patented process is used outside of Prime Focus. In a 2011 interview, Dr. Barry Sandrew, founder of Legend3D, describes a general process for 2D to 3D conversion:
We have a proprietary process we developed for colorization which we simply call masking. Those masks were originally used to identify where and what certain colors are designed to be. In 2D to 3D conversion, rather than applying color using these masks, we apply depth. We developed a whole new process for converting these masked frames into 3D in a unique way…
Dr. Sandrew goes on to describe the difficulty of the conversion and how it “requires an army of artists with a lot of talent,” so there are a “few studios like Legend3D who can convert an entire feature film but it is less than a handful.”
If the Process is So Difficult, Why Not Just Shoot the Movie in 3D?
Sometimes, when first hearing about 2D to 3D conversion, people ask why the film is not shot with stereo cameras in the first place, thereby eliminating the need to “convert” to 3D—i.e., wouldn’t it make more sense to create a stereo image pair during filming than to mimic the change in perspective? Other critics of 3D-conversion are familiar with a 2010 movie called Clash of the Titans, which, despite grossing close to a half-billion dollars globally, prompted director Louis Leterrier to say the 3D conversion was “famously rushed and famously horrible.” Regardless the reason, a vast majority of people come into the conversation with the notion that “real” 3D is better than 3D conversion. However, they might be missing half the picture.
Wanting to film the original in 3D rather than convert may be a valid thought, and many instances are still shot in 3D, however there are several downsides to filming in three dimensions, which are typically expense-related or artistically driven. For instance, not only would a 3D camera rig require an additional camera and gyroscopes, the post-production workflow is centered on 2D images. Any disturbance (or doubling) of the complex processes of filming and editing will necessarily mean more time and money. (More here)
Artistic reasons for filming in 2D may depend on the comfortability of the filmmaker with the medium. Considering that most people will see the film as a 2D version, arguably, catering to the 3D effects may overwhelm the viewer or detract from the story. Furthermore, shooting in 3D can create problems with reflections and glares from different angles, as well as some inherent restrictions on zooming and proximity of the cameras to the action.
There are only a handful of firms that appear to frequently work with feature film 3D conversion: Prime Focus, Legend3D, Cinesite, Stereo D, In-Three/Digital Domain and Gener8. Other companies, such as 3D Eye Solutions and Plantation Village Studios appear poised to take on feature films, and list contributions on their websites.
Looking at value of the defendant company, Legend3D raised $19 million of funding in 2011 and another $8 million in 2013. On January 7, 2015, Legend3D announced four unnamed feature film conversion projects through summer 2016. Legend3D is planning to utilize their 400+ artists and production staff between their Hollywood and Canada offices, along with their “new proprietary depth grade node and real-time conversion pipeline.”
As an example of the revenue generated by the converted films, Legend3D supported a re-release of Top Gun in 2013 which saw $1.97 million at the IMAX box office the opening weekend and $1.43 million in Blu-ray sales the first week after release. Then again, The Amazing Spider-Man 2, which Legend3D supported with 3D effects and conversion, brought in $91.6 million on opening weekend last May.
Such conversions can be relatively expensive, with the conversion by Stereo D of Titanic costing $18 million and 60 weeks. Apparently, satisfying the fastidious James Cameron was worth it, as the 3D Titanic made more than $25 million over its first five days of release in 2012. However, profit margin for video effects and conversion companies does not appear large, as Legend3D founder Dr. Sandrew paints an industry ripe for better pipelines and workflow streamlining as we even move closer to 4K Ultra High Definition as a baseline.
With regard to the films themselves, according to Mark Hughes of Forbes (November 16, 2014):
3D films comprised 12 of the top 13 highest-grossing films of this year  so far, with those films amassing a huge $7.5+ billion and counting at the worldwide box office. The rest of 2014?s 3D release have likewise contributed an additional $1+ billion to date, with several major 3D release still to come that should push the finally 3D box office tally over $10 billion and likely toward the $12 billion mark.
Many 2013 and earlier articles proclaimed the 3D “fad” at theaters dead, but from the figures above, it remains to be a draw at home and abroad. Audience fatigue may set in, but one can hardly picture a studio releasing a movie featuring superheroes, aliens, or robots-in-disguise merely in two dimensions. For many studios, directors and projects, 3D technology is a necessity.
To further complicate the economic considerations of the 3D world, RealD Inc., maker of 3D film projectors, optical components and cinema systems, appears to be interested in a potential sale or merger. According to a February Deadline.com article, RealD rejected an unsolicited $600 million offer and is seeking “strategic alternatives.” RealD’s CEO told analysts that “Demand for 3D remains strong in key markets” including China, as they plan to show 3D releases of The Avengers: Age Of Ultron, Jurassic World, the final Hunger Games and Star Wars: Episode VII – The Force Awakens on their 25,000 screens this year.
While the popularity of 3D movies at the box office may ebb and flow, with a 2010 estimate of the conversion process costing $50,000 – $100,000 per minute of material, the 2D to 3D conversion process can certainly remain high in revenues, if not profitable. Even if in the near future all the studios were to switch to filming in “real,” native 3D, post-production conversion will be necessary for long shots, zooms, CGI effects, and particles like smoke. In what appears to be a highly competitive market for quality feature-length 3D conversion—where fighting for one 90-minute film’s work at a time, or even sharing a Transformers movie, may be the norm—a patent lawsuit may be a market competitor’s only chance at getting ahead.
3D Conversion Patent Disputes of the Past
If the monetary stakes weren’t exciting enough, previous litigation and licensing around the technology indicate this dispute will be high profile news. In 2011 Digital Domain Media Group sued Prime Focus for infringement of six patents that DDMG acquired in the 2010 purchase of stereo studio In-Three. At the time of the acquisition, DDMG had 3D-post-production success from Disney’s TRON: Legacy, but added “pioneering” 3D conversion technology from In-Three, which was best known for its work with Tim Burton’s Alice in Wonderland.
In July 2012, the two parties came to an undisclosed settlement where DDMG licensed its 3D conversion technology to Prime Focus World, and agreed to collaborate on visual effects and 2D-to-3D conversion services work for feature films. While the six In-Three patents are no longer in the spotlight, the patent-at-issue in the current Legend3D suit references a number of patented inventions by Michael Kaye.
Later in July 2012, in an investor call, when asked, “Who is infringing [DDMG’s patents]?” DDMG Chair and CEO John Textor told investors, “Everybody that has an interest in a 3D film … from the content creators to the production companies to the distributors, the exhibitors, the projector companies … the theaters, down to the television sets and the videogames.” Textor identified RealD and IMAX as additional targets for lawsuits and it appeared that DDMG’s licensing strategy had only begun as Samsung and Reliance Media Works quickly followed by signing licensing deals.
The 3D and video effects industry was abuzz, but the talk of DDMG putting the squeeze on Hollywood simmered when Digital Domain Media Group filed for bankruptcy in September 2012 and sold their 3D Patents to RealD for $5.45 Million. Disney filed an objection to the purchase, claiming that agreements it had with DDMG (and In-Three) created “an unlimited, non-exclusive, perpetual worldwide license” for its use of the technology. The bankruptcy court disagreed, only allowing Disney a limited license to the patents in its films G Force and Alice in Wonderland. The Third Circuit decided the appeal based on contract law, though Disney apparently based its claim in patent law (i.e., that a covenant not to sue is a license, and that the buyer of a patent takes the patent subject to pre-existing licenses).
DDMG is quieter now, refers to itself only as Digital Domain, and was a producer on Ender’s Game. Many people are still waiting on Digital Domain’s 2012 plans for 3D hologram versions of Tupac and Elvis to go on tour.
Stereoscopynews.com speculated that Disney’s worry about patent suits has led to a halt on 3D conversions so much that they canceled a 2013 release of The Little Mermaid 3D (Legend3D evidently did some conversion work for this). Part of the logic might be that while The Lion King 3D conversion grossed nearly $95 million, other converted films such as Beauty and the Beast 3D ($47.6MM), Finding Nemo 3D ($41MM) and Monsters, Inc. 3D ($33 MM) did not perform as well. However, Disney—who now owns Lucasfilm as well the Marvel movie rights—appears to have little qualms with releasing 3D movies such as Guardians of the Galaxy, Avengers, and the December-release Star Wars: The Force Awakens.
The Next Dimension
In an April press release, Brian Robertson, current CEO of Legend3D, said the Prime Focus World allegations are “without merit” and that “Legend3D’s patent portfolio predates the single Prime Focus patent by eight years.” Mr. Robertson also said that “Legend3D’s proprietary and patented conversion process is fundamentally different than that claimed in the Prime Focus patent.” Even without analyzing their 12 U.S. patents, it would not be a surprise to anyone to see Legend3D leverage their IP assets Prime Focus, either in litigation or licensing. In fact, brand new U.S. Patents 9,007,365 and 9,007,404, titled “Line depth augmentation system and method for conversion of 2D images to 3D images,” and “Tilt-based look around effect image enhancement method,” respectively, issued to Legend3D on April 14, 2015. Both cite PFW’s ’628 Patent’s earlier application publication, but Legend3D’s ’365 Patent appears more relevant as it describes a method of 3D conversion via adding depth to regions by altering depth of lines in the regions.
At this point, any speculation on the outcome of the trial is complete conjecture. Obviously, the above-analysis lacks insight into each team’s strategies and is with little evidentiary basis. However, Prime Focus would likely not risk filing suit if their attorneys did not perceive this to have merit. Perhaps their team has developed simpler methods of identifying potential infringement beyond in-depth, frame-by-frame analysis of stereo image pairs. Who knows, maybe there’s a smoking gun from when each company was doing their respective work for Transformers? Probably not. Like most patent infringement cases, this one may come down to the judge’s claim term constructions and the testimony of each party’s expert witnesses—if it even gets to that level.
All that can be said for certain is that this narrative is still in its expository portion. But, based only on knowing the market and the fallout from previous 3D-conversion patent litigation, I know I will not be the only one watching this blockbuster.