[ad_1]
Final week, MBW printed a extensively shared report about TikTok and its mum or dad ByteDance‘s ambitions within the discipline of AI-generated music.
Inside that article, we instructed you concerning the intensive analysis carried out by ByteDance’s Speech, Audio & Music Intelligence (SAMI) crew within the discipline of AI music.
A kind of analysis initiatives describes a generative mannequin known as MeLoDy that was educated, in accordance with ByteDance’s SAMI researchers, on 257,000 hours of music (learn the paper right here) that got here from 6.4 million audio recordsdata.
We additionally instructed you about two US patents, secured by ByteDance in current months, that present IP safety within the US for the Chinese language firm’s AI music-related applied sciences.
Now, we’ve uncovered an further batch of AI music-related patent and trademark filings from ByteDance that can increase quite a lot of eyebrows within the music trade – overlaying subjects together with “information mining” and “music technology”.
Apparently, these US patents and trademark purposes had been filed by a mysterious ByteDance affiliate known as Lemon Inc.
There’s not numerous details about Lemon Inc. on-line. We do know, nevertheless, that (a) it’s included within the Cayman Islands, and (b) It’s been very busy over the previous couple of years submitting purposes to the USA Patent and Trademark Workplace. (It even utilized for a e book publishing-related trademark firstly of 2023).
Lemon Inc. isn’t talked about particularly on ByteDance’s company construction diagram (see under, and right here).
Nevertheless, the World Mental Property Group, through its Arbitration experiences, not too long ago described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Web know-how firm that gives cellular purposes (“apps”) akin to TikTok”.
Lemon Inc. is to not be confused with one other ByteDance product known as Lemon8, the Instagram/Pinterest rival launched within the US final yr, which some speculate is being pushed out there as a “back-up” within the occasion of a possible TikTok ban.
As reported by Axios final yr, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered on the similar deal with as TikTok in Singapore.
Why Lemon Inc. is getting used as a automobile to use for emblems and patents in the USA on behalf of ByteDance is a query for an additional day.
For now, feast your eyes on what else ByteDance has been as much as…
Lemon Inc. PAtent 1: ‘Automated and quick technology of music audio content material for movies’
In what affords the clearest glimpse at what ByteDance may be planning to make use of its AI music-related tech for, its affiliate Lemon Inc owns a patent in the USA for a “methodology of mechanically and effectively producing music audio content material for movies”.
The patent software was filed in July 2022, granted in September 2023, and is assigned to each Lemon Inc. and ByteDance (see under).
Based on the submitting, which you’ll be able to learn for your self right here, this patent covers “strategies that allow environment friendly sensible music technology {and professional} music enhancing”.
“After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video will likely be mechanically generated. Such music or audio could also be royalty-free and may be of any size.”
Excerpt from patent
The submitting provides: “After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video will likely be mechanically generated. Such music or audio could also be royalty-free and may be of any size.”
The submitting explains additional that the tech can extract “info from the video, whereby the extracted info includes info indicating movement velocity within the video, info indicating movement saliency within the video, info indicating scene transition within the video, and timing info related to the video”.
It could then generate a “plurality of units of music notes matching the plurality of segments of frames primarily based at the least partly on the extracted info utilizing a mannequin, whereby the mannequin is pre-trained and learns to correlate video movement velocity with music notice density, correlate video movement saliency with music notice energy, correlate video scene transition with music construction, and correlate video timing with music beat”.
Simply to additional spotlight the shut hyperlinks between Lemon Inc, TikTok and ByteDance, right here:
Amongst the named inventors of this tech within the patent submitting are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who in accordance with LinkedIn are a Senior Analysis Engineer at TikTok, and ML Analysis Engineer at ByteDance, respectively.
Lemon Inc Patent 2: Computing system and methodology for music technology
Lemon Inc additionally has one other patent pending for a “Computing system and methodology for music technology”.
This patent was filed by Lemon Inc USA in June 2022 and assigned to each Lemon Inc and ByteDance in September 2023 (see under). It was printed in December.
The submitting explains that “Packages have been developed that may generate music primarily based on a lyric inputted by a consumer” however that the music generated by such packages “usually lacks musical qualities that many individuals respect, and thus isn’t very song-like”.
It continues: “For instance, auto-generated music from such packages can undergo from misalignments in lyrics and melody notes, scattered or disjointed group and tune construction, mismatched rhythm tracks, and lack of a catchy repeating melody.
“Because of this, such packages haven’t achieved widespread use. Because of this, a barrier presently exists to speedy tune improvement utilizing such packages.”
The patent software, which you’ll be able to see right here, describes “a music technology system” that options “a processor and a reminiscence operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.
The invention additionally includes “a music technology program saved within the reminiscence and executed by the processor to be configured to obtain a consumer enter of lyrics, determine a plurality of syllables within the lyrics, decide a syllable sample within the recognized plurality of syllables, match the syllable sample to a specific rhythm template of the plurality of rhythm templates”.
The invention is then proposed to “generate a melody primarily based on the chosen rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.
The diagram under units out the way it works:
Lemon Inc.’s Logos
Along with its energetic and pending patents within the discipline of AI Music, Lemon Inc. has additionally filed for a number of emblems in the USA that confer with AI, music technology and music tech generally.
A kind of emblems listed on the USA Patent and Trademark Workplace (USPTO) is for a service known as ‘BytePlus‘ which seems to have been formally registered in Could 2023.
TikTok and Lemon Inc mum or dad firm ByteDance launched BytePlus in 2021 reportedly as a brand new division to promote its AI know-how. ‘BytePlus’ is listed in ByteDance’s company construction diagram.
The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the submitting states are “pc software program platforms for producing and delivering personalised suggestions to prospects within the fields of synthetic intelligence, information mining, information analytics, and machine studying covers a variety of AI-related actions.
The trademark additionally covers “machine-learning primarily based music technology, and machine language translation; software program consultancy providers regarding synthetic intelligence, information mining, information analytics, and machine studying”.
Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio alerts after which “re-renders” these alerts utilizing what it says is machine-learned fashions of musical devices.
The Mawf trademark was utilized for in July 2022 and formally registered by the USPTO in September.
The submitting covers “Laptop software program, particularly, downloadable software program for creating music movies, recorded music composition software program”.
Lemon Inc. has additionally filed (in Could 2023) two purposes to trademark logos for what the filling seems to explain as a music-making app (see under). These logos resemble the logos of ByteDance’s Ripple app.
The appliance described within the pair of filings, which seem like beneath overview at present, cowl “Downloadable software program for processing music and audio recordsdata; Downloadable music-composition software program; Downloadable software program for creating, recording, and enhancing audio to create music and audio; downloadable software program for creating music and audio; Downloadable cellular software software program enabling customers to share music and audio they created for different customers to view, entry, and remix throughout the software; Downloadable software program to be used in music and audio manufacturing“.
Each emblems for various coloration variations of the emblem seem to have been filed in April and Could 2023 and have been energetic as of January this yr.
Lemon Inc. additionally filed a trademark software for ‘Ripple‘ in February 2023 and the mark was solely simply printed on the USPO’s Trademark Official Gazette final month.
Ripple is the AI-powered music-making app launched by ByteDance in June final yr that may flip a hummed melody right into a tune.
Apparently, Lemon Inc additionally filed a trademark software again in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.
SoundOn lets artists add their music on to TikTok and Resso, and it might probably additionally distribute artists’ music to platforms like Spotify and Apple Music.
And simply in case you’re nonetheless not satisfied about Lemon Inc’s affiliation with TikTok mum or dad ByteDance, Lemon Inc. utilized for a trademark for “ByteDance” in January.
MBW’s reporting on ByteDance’s and TikTok’s AI-related music actions in the USA arrives in opposition to a backdrop of two large ongoing tales involving the corporate and its subsidiary TikTok.
The primary story is a looming TikTok ban in the USA, because of the potential compelled sale of its US operations through a invoice handed via the Home of Representatives final week, which is now one giant step nearer to changing into legislation.
The opposite story that we’ve been following intently is TikTok’s falling out with Common Music Group, which refuses to resume its licensing cope with TikTok for quite a lot of causes, together with the app’s compensation for artists and songwriters, and UMG’s considerations about TikTok’s use of AI.
On March 1, Common Music Publishing’s catalog of ~4 million songs grew to become unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (thus far with out renewal) on February 1.
In a assertion issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the function AI-generated audio is taking part in on TikTok.
UMPG claimed that, thus far, TikTok has not supplied Common with any assurances that the platform gained’t practice its AI fashions on the music firm’s songs.
MBW has been writing for a while now concerning the hypothetical potential for TikTok and different providers to fill their catalogs with music generated by AI, which might, within the course of, dilute the market share of conventional rightsholders.
In February final yr, we printed an ‘MBW Reacts’ article asking if TikTok might pull off a “heist” on the music trade on this regard, following its aggressive funding in generative AI know-how.
The “heist” we had been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to properly over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.
We wrote: “With music taking part in such a key function in TikTok’s rise, if main label content material does disappear from the platform – and the hole is one way or the other efficiently crammed by indie and AI-driven creations – TikTok could possibly be stated to have pulled off one of many greatest heists in music enterprise historical past. A bait and swap for a billion customers.”Music Enterprise Worldwide
[ad_2]
Source_link