Skip to content
DnsLister Forum

Where domain hunters compare notes

TikTok bans scraping in their ToS and some guy just posted 5.6B videos on HuggingFace

I was looking for some agent models on Hugging Face, then after tinkering around I went to check whats new in the dataset tab. To my surprise some dev dumped metadata for about 5.6 billion TikTok videos on Hugging Face. It includes captions, hashtags, sound IDs, shop product and seller IDs, every engagement count. 460GB of data. Now, to my knowledge, TikTok does not allow scraping of their videos "legally", according to the devs own explanation they went straight at the private mobile API with generated device identities that pass as Android phones, reverse engineered request signatures and a spoofed TLS handshake. What that means is that no account and no login was required. Can you believe this? They claim 3 billion profiles, almost 6 billion videos and 2.8 billion comments in THREE WEEKS. Meanwhile what I've seen on this sub in particular, probably few others too, but this one in particular is that people here spends half if not more time arguing about whose IP pool is cleaner. And TikTok is one of those more difficult targets to scrape. What I'm trying to say is that nobody built this by being clever with IPs, checking whether they leak DNS and other data, whether they are "clean" or not abused or whatever, they built it by making the approach look like a legitimate phone. I mean, that is just crazy. Anyways, I saw that the free version is non commercial only, the dataset I mean, if you want creator profiles, daily updates basically run the whole thing yourself, or to use it for anything that makes you money, you should visit their site the code is up there for $1,699.

Now why this caught my interest in the first place. TikTok's terms ban this kind of scraping, and the official research API is mostly for academics in a few regions. Reddit is a good contrast, since it's taking scraping to court with the Perplexity case and the judge let most of it go forward. With this dataset I haven't seen anything like that. It's just there on HuggingFace with a free download link, and I haven't found anything on the web that TikTok is doing anything about it. Maybe that will soon change.

Source: r/ProxyEngineering · by /u/elgordooo17

Leave a Reply

Your email address will not be published. Required fields are marked *