Data collection for distant reading

Data collection for distant reading#

Automated data collection#

This repository has a focus on automated data collection with Python. To learn more about collecting podcast reviews and YouTube comments, please read the relevant sections under Skills (see page menu). At Maastricht University, we use our institutional Data Science Research Infrastructure (DSRI) to run code. This process is also explained under Skills. If you want to run code in a different environment, you may have to adjust the scripts accordingly. Please ask an expert for help if needed.

Research strategies for data collection#

Finding sufficient data for a topic you are interested in can be challening, also because relevant data may not include the most obvious keywords. Here is some advice to identifying useful data sets:

Finding relevant search terms#

Regardless of the search engine or platform you use to find content related to a topic, it is important to carefully develop your search terms. Mistakes that students often make is that their search terms are either too broad or much too narrow. For example, searching for a podcast or video on “history” will give you more results than you can process, and it will not help you to create or answer a clear research question. Instead of searching for “history,” you may want to search for “medieval history in France” to get a more manageable list of results. One example of the opposite challenge is that a student of mine in a past course tried to find podcasts on “genital mutilation” but the few results that came up had absolutely no reviews they could analyse. In such cases, it is recommended to branch out and consider related aspects such as female bodies, female sexuality, or violence against women.

Use platform-integrated search features#

When searching for videos or podcasts to scrape user comments, you can, of course, begin your search with the platform-integrated search features. Most people will find this easy on YouTube, but not everyone may be familiar with the App Store. Here, too, you can directly open the Apple podcasts app on your device or use the search box on the website for research, but you should be aware that reviews may only be visible when you open the show / episode website in your browser. This is a special challenge for MAC users as their machines tend to automatically re-direct to the app. Also, you have to consider that searching by keywords in the Apple Store will only give you limited results because the search engine focuses on podcast titles and the short podcast descriptions. If your keyword is not mentioned, a relevant podcast may not come up as a result, so using external research tools is recommended.

Aggregators, directories and recommendation sites#

Websites like Podchaser, FeedSpot, or Listen Notes list top-rated and specialised podcasts. You can often filter podcasts by subject, reviews, and popularity. In addition, blogs and (tech) news sites also regularly feature podcast recommendations, such as the 60 Best Podcasts (2024) presented by online magazine Wired. After finding relevant podcasts, check whether they are available on Apple Podcasts. Most professional podcasts are on Apple. Similar services can also be used to identify popular YouTube channels or videos with many comments. Tech sides like Medium give advice on topics like “9 Ways To Find Trending Topics On YouTube In 2024”.

Explore social media and discussion forums#

Platforms like X, Mastodon, Threads, BlueSky, Reddit, and Facebook groups often have communities of video fans and podcast listeners who discuss and recommend shows. Subreddits like r/podcasts can be a great source for finding highly recommended or niche podcasts. You can also consider asking other social media users for their podcast or video recommendations on a specific topic.