On June 22, 2021, a Western District of Texas jury returned a verdict in Freshhub, Inc. v. Amazon.com Inc., finding that Amazon’s Alexa devices do not infringe three voice technology patents owned by Freshub. We want to report and discuss this case on the AI Patent Blog because virtual assistant software and products represent a commercially significant form of AI technology, and this case dives into the nuanced differences between legacy voice recognition software and modern AI-based virtual assistants.
By way of background, Plaintiffs Freshub, Inc. and Freshub, Ltd. (collectively “Freshub”) filed a complaint against Amazon.com, Inc., Amazon Digital Services, LLC, AmazonFresh, LLC, Prime Now, LLC, and Whole Foods Market, Inc. (collectively “Amazon”) on June 24, 2019, alleging Amazon’s Alexa devices infringe U.S. Patent Nos. 9,908,153 (“the ’153 patent”); 10,213,810 (“the ’810 patent”); and 10,232,408 (“the ’408 patent”) (collectively, “the Asserted Patents”). The Asserted Patents are generally directed toward a voice processing system to collect, manage, and display items described by a user. Asserted Claim 1 from the ’153 patent is reproduced below for reference:
1. A voice processing system comprising: a first system configured to receive user spoken words comprising: a microphone; a wireless network interface; a digitizer coupled to the microphone, wherein the digitizer is configured to convert spoken words into a digital representation; a first computer; non-transitory memory that stores instructions that when executed by the first computer cause the first system to perform operations comprising: receive via the digitizer a verbal order, comprising at least one item, from a user, wherein the verbal order was captured by the microphone and digitized by the digitizer; immediately transmit, using the wireless network interface, the digitized order to a computer system remote from the first system; the computer system, the computer system comprising: a networks interface; a second computer; non-transitory memory that stores instructions that when executed by the second computer cause the computer system to perform operations comprising: receive, using the network interface, the digitized order from the first system; translate at least a portion of the digitized order to text; identify an item corresponding to the text; add the identified item to a list associated with the user; enable the list, including the identified item, to be displayed via a user display.
Freshub alleged that Amazon’s accused Alexa products infringed the Asserted Patents by utilizing Alexa to take spoken words and translate those words to add items to an Alexa Shopping List associated with the user’s Amazon account, with the added items being viewable to the user by web browser of an Amazon App Product. Amazon responded to Freshub’s complaint asserting defenses on non-infringement and invalidity.
Amazon’s Alexa works by having a wake word (“Alexa”) that activates the device and starts recording the user’s request. When the user finishes speaking to the Alexa, the device sends a recording of the request over to Internet to Amazon. Amazon then processes this recording through their service called Alexa Voice Services. This service converts that recording into commands that the user’s Alexa device can interpret. So for example, if a user says “Alexa, add bread to my shopping list,” that recording is sent to Amazon and converted by Alexa Voice Services into a command Alexa can understand.
At trial, Amazon’s noninfringement arguments focused on three points. First, Amazon argued that its Alexa product did not meet the “digitized order” limitation, arguing that the limitation required the digitized order to be received before any translation of the spoken words to text even takes place. Amazon maintained that because a user can say anything to Alexa, Alexa must first convert the user’s utterances to text and only once the utterance has been converted to text can Alexa determine the user’s intent, meaning that, when the utterance is made, Alexa has no information about whether the utterance is an order, request for a joke, or a command to play music.
Second, Amazon argued that its Alexa product did not meet the “identifying an item corresponding to the text” limitation of the asserted claims. Amazon argued that when a user adds to his/her Alexa Shopping List, the words uttered by the user appear on the list, but those entries are not actually matched up with an actual item corresponding to the utterance.
Finally, Amazon argued that its Alexa product did not meet the “match…text…to a text description” limitation of the ’408 patent or the “use the text…to identify an item corresponding to the text description” limitation of the ’810 patent. Amazon argued that the claim limitations require the instrument control electronics to use the text output from the automatic speech recognition component to match or identify an item based on a text description. Amazon argued that rather than using the text output to identify items, Alexa used the intents created by Amazon’s NLU (natural language understanding) software to search for relevant products for the user.
Amazon also argued that the Asserted Patents were unpatentable under 35 U.S.C. § 101, and invalid under 35 U.S.C. §§ 112 and 103. While the jury verdict found the three Asserted Patents not to be invalid, it did not find that Amazon’s Alexa infringed any of the three Asserted Patents.
It is worth taking a big picture look at where the jury seemed to think the claims of Freshub’s patents and Amazon’s Alexa technology diverge. The jury may have believed that the technology the patent was based on is representative of a system where a dialog and process follow a predetermined script to perform one or more specific actions, while Amazon’s Alexa is capable of performing a multitude of user services and is therefore acting without a script. This, coupled with the underlying fact that Freshub’s patents were based on a 2005 patent filing, a time when AI virtual assistants and natural language understanding was in a very different place than it is today, may have ultimately led the jury towards its non-infringement verdict, especially considering how much emphasis Amazon’s counsel placed on this point during closing arguments. While we will never definitively know which of Amazon’s non-infringement arguments won the day, it’s important to view these arguments in light of how much AI technology has changed over the past 15+ years.