Skip to content

Instantly share code, notes, and snippets.

@techykajal

techykajal/total_tokens.py

Created August 28, 2021 13:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save techykajal/7b0538a21726d52709ea554136b73044 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save techykajal/7b0538a21726d52709ea554136b73044 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
total_tokens.py
# Total no. of tokens in the corpus
tokens = []
for article in DF['Updated_content']:
for word in article:
tokens.append(word)
len(tokens) # There are total 1 lac 50 thousand tokens in the corpus
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment