The following URL (Uniform Resource Locator) will be used to show web scraping using Python.

• CNN RSS (Really Simple Syndication) page at https://www.cnn.com/services/rss/#terms (we will use just one feed for example purposes)

First, some preliminaries. It is often necessary to do tasks and save data on a periodic basis. Rather than use a full database system, which incurs a huge overhead in complexity, a more no-SQL approach using a file system can be used.

A CNN RSS feed will be used as an example RSS feed. Image as of 2020-03-23.

This appears to say that you may not use any of the CNN RSS feeds to compete with CNN. Image as of 2020-03-23.

This appears to say that you may not use any of the CNN RSS feeds to compete with CNN. Image as of 2020-03-23.

The term scraper is here used in the general sense of a scraper.

In terms of the above restrictions, Python is here being used to create a personal custom RSS reader and processor for personal educational and research use.

At no time should one try to use techniques shown here for commercial use in any way without getting proper consent and legal advice!

Note: The Python program(s) shown here are toy programs to illustrate concepts. Some work would be required to make them generally useful for sophisticated personal educational and research use.

The first step would be to grab the current RSS feed and save it for later processing. In general, a data flow pipeline is used for the entire system (to be discussed later).

This requires just one web access to get the RSS feed one time and save it as text such that it can be opened (e.g., using Notepad++) to inspect and determine how to further process it.

A simple RSS feed list can be represented as follows, where only one item is used.
rssFeedList1 = [ ["CnnTopStories", "http://rss.cnn.com/rss/cnn_topstories.rss"], ]

Note: In these program examples, the RSS feed list is repeated. An enhanced system would use a module (or JSON file) for the list and update functions - or even a class.

Note: In an enhanced system, a (random or uniform) time delay might be added between requests.

Today is used for processing as follows.
import datetime import time dashDateFormat1 = "%Y-%m-%d" dashTimeFormat1 = "%H-%M" today1 = datetime.date.today() dashDate1 = today1.strftime(dashDateFormat1)
Here is the Python code [#3]
import os import urllib.request import datetime import time dashDateFormat1 = "%Y-%m-%d" dashTimeFormat1 = "%H-%M" today1 = datetime.date.today() dashDate1 = today1.strftime(dashDateFormat1) rssFeedList1 = [ ["CnnTopStories", "http://rss.cnn.com/rss/cnn_topstories.rss"], ] print("Get RSS feeds for {0:s}:".format(dashDate1)) for rssName1,rssUrl1 in rssFeedList1: rssFs1 = "./{0:s}-{1:s}.rss".format(rssName1,dashDate1) print("") print("\tName: \"{0:s}\"".format(rssName1)) print("\tFile: \"{0:s}\"".format(rssFs1)) if os.path.isfile(rssFs1): print("\t\t(file exists , nothing done)") else: try: print("\tURL: \"{0:s}\"".format(rssUrl1)) urllib.request.urlretrieve(rssUrl1,rssFs1) # sleep one second time.sleep(1) except Exception as e: print("\tError getting request:")
Here is the output of the Python code.
Get RSS feeds for 2021-03-22: Name: "CnnTopStories" File: "./CnnTopStories-2021-03-22.rss" URL: "http://rss.cnn.com/rss/cnn_topstories.rss"
Note: Most RSS files are huge and not easily read with a text editor such as Notepad++.

Luckily an RSS file has a fairly well-defined format and can be parsed as XML (Extensible Markup Language).

Note: The urlretrieve function combines the steps of opening the URL, reading the response, and saving the response text (as binary).

Next, a separate program is used to process the feeds.

A general program (omitted) might process all feeds for all dates, or just the dates that have not yet beet processed (in the data flow model used).

In an enhanced system, folders might be used, one per RSS feed name, in which case the RSS feed name would not be needed in the file name, just the date.

Here is the Python code [#4]
import os import re import feedparser import datetime import time dashDateFormat1 = "%Y-%m-%d" dashTimeFormat1 = "%H-%M" today1 = datetime.date.today() dashDate1 = today1.strftime(dashDateFormat1) rssFeedList1 = [ ["CnnTopStories", "http://rss.cnn.com/rss/cnn_topstories.rss"], ] print("Process RSS feeds for {0:s}:".format(dashDate1)) for rssName1,rssUrl1 in rssFeedList1: rssFs1 = "./{0:s}-{1:s}.rss".format(rssName1,dashDate1) print("") print("\tName: \"{0:s}\"".format(rssName1)) print("") if not os.path.isfile(rssFs1): print("File: {0:s} (not found)".format(rssFs1)) else: (_, _, _, _, _, _, rssSize1, _, rssTime1, _) = os.stat(rssFs1) print("File: {0:s} ({1:,} bytes)".format(rssFs1,rssSize1)) feed1 = feedparser.parse(rssFs1) title1 = feed1["feed"]["title"] print("") print("Title: \"{0:s}\"".format(title1)) print("") print("Keys: (first entry)") entry1 = feed1.entries[0] for key1 in entry1: print("\t{0:s}".format(key1)) for entry1 in feed1.entries: title2 = entry1["title"] print("") if "published" in entry1: published2 = entry1["published"] print("{0:s}".format(published2)) else: print("(published missing)") print("{0:s}".format(title2)) link2 = entry1["title"] if "summary" in entry1: summary2 = entry1["summary"] try: summary3 = re.sub(r"\<.*?\>"," ",summary2) summary3 = re.sub(r"\s+"," ",summary3) print("{0:s}".format(summary3)) except Exception as e: print("(bad characters in summary)") else: print("(summary missing)")
The feedparser package will be used.

One way to install the feedparser package is as follows.

Use the path to your installed version of Python. Here is the output of the Python code.
Process RSS feeds for 2021-03-22: Name: "CnnTopStories" File: ./CnnTopStories-2021-03-22.rss (255,506 bytes) Title: "CNN.com - RSS Channel - HP Hero" Keys: (first entry) title title_detail links link id guidislink published published_parsed summary summary_detail feedburner_origlink Mon, 22 Mar 2021 10:41:12 GMT A generation wants change. But witness accounts and leaked police video reveal a brutal crackdown with beatings and rape. Mon, 22 Mar 2021 09:18:07 GMT Leaked video from inside police stations shows ferocious beatings As part of a months long investigation into the crackdown inside Belarus, CNN has spoken to several Belarusians who fled the repression of President Lukashenko's regime. In dozens of interviews, protesters and opposition activists have spoken of torture -- from systematic beatings to rape with a police baton. CNN's Nick Paton Walsh reports. Sat, 20 Mar 2021 14:15:48 GMT Biden administration off to a combative start with tough rhetoric on Russia, China President Joe Biden didn't waste a beat when asked this week whether he believed his Russian counterpart Vladimir Putin was a "killer." Mon, 15 Mar 2021 21:50:30 GMT Rights groups file landmark legal case against Russian Wagner mercenaries Advocacy groups from Russia, France and Syria have filed a legal case against Russian mercenary group Wagner, accusing its fighters of torturing and beheading a Syrian man in 2017. Wed, 17 Mar 2021 12:41:01 GMT Biden says Putin 'will pay a price' for Russian efforts to undermine the 2020 US election President Joe Biden said Vladimir Putin "will pay a price" for his efforts to undermine the 2020 US election following a landmark American intelligence assessment which found that the Russian government meddled in the 2020 election with the aim of "denigrating" Biden's candidacy. Thu, 18 Mar 2021 19:08:02 GMT Hear Putin's response to Biden calling him a 'killer' Russian President Vladimir Putin responded to President Joe Biden comments made during an ABC News interview in which he described Russian President Vladimir Putin as a "killer." Tue, 16 Mar 2021 10:47:05 GMT First Russia poisoned him. Now this is the prison camp for Alexey Navalny At the rusty gates of Penal Colony No. 2, dour prison officers in military fatigues turn unwanted visitors away. Mon, 22 Mar 2021 12:06:15 GMT AstraZeneca touts 79% efficacy rate for vaccine and prepares to seek emergency use authorization • Where do experts stand on the safety of the AstraZeneca vaccine? • Miami Beach officers shoot pepper balls into spring break crowds Sat, 20 Mar 2021 17:43:24 GMT Reporters call out Biden admin for lack of transparency at the US-Mexico border • Border Patrol numbers show another jump in children held for more than 10 days • Dana Bash pushes Homeland Security Secretary on timeline for suitable border facilities Mon, 22 Mar 2021 01:03:32 GMT Rep. Tom Reed takes 'full responsibility' after allegation of sexual misconduct Republican Rep. Tom Reed of New York apologized and took "full responsibility" on Sunday for the sexual misconduct allegations made against him by a former lobbyist. Mon, 22 Mar 2021 08:42:05 GMT Trump is returning to social media with his own platform, spokesman says • Defense Secretary Lloyd Austin travels to Afghanistan on unannounced visit • Opinion: The shameful vote by 172 Republicans Mon, 22 Mar 2021 12:48:45 GMT Olympic surfing hopeful Katherine Diaz, 22, killed by lightning while training A 22-year-old Olympic surfing hopeful from El Salvador has died while training for an upcoming qualifying competition, the country's National Institute for Sport (INDES) has confirmed. Sat, 20 Mar 2021 01:24:00 GMT Wisconsin voters weigh in on Sen. Ron Johnson CNN's Sara Murray takes a look at Sen. Ron Johnson (R-WI) and how voters in the state feel about his comments surrounding the riot at the Capitol, and the possibility that he will run for reelection in 2022. Mon, 22 Mar 2021 12:15:21 GMT Putting 'cologne on Jim Crow': Georgia GOP lawmakers drive toward new voting restrictions Republican lawmakers in Georgia are racing to pass sweeping bills that would clamp down on ballot access and give legislators new powers over elections, following record voter turnout in 2020 that helped Democrats seize the White House and control of the US Senate. Sun, 21 Mar 2021 15:36:59 GMT Brian Stelter: This Fox News host doesn't book experts to answer his questions CNN's Brian Stelter looks at the media's coverage of vaccinations and vaccine hesitancy. Sun, 21 Mar 2021 16:02:54 GMT Analysis: Cuomo hasn't resigned because he still has support from Democratic voters New York Gov. Andrew Cuomo is fighting to stay in office. He's resisting calls from several Democrats to resign after multiple women have come forward with allegations of sexual harassment. Sun, 21 Mar 2021 19:03:10 GMT Two hikers found dead in Acadia National Park after cliffside fall Two people were found dead in Acadia National Park in Maine on Saturday after a fatal fall during a hike, according to the National Park Service. Mon, 22 Mar 2021 10:00:37 GMT Biden moves rapidly to shore up Obamacare and erase Trump's changes President Joe Biden is wasting no time establishing his vision for the Affordable Care Act and reversing many Trump-era measures aimed at weakening it. Sun, 21 Mar 2021 15:19:37 GMT She lost her husband to cancer. Now, her obituary for her 'dead sexy' spouse has gone viral Crystal Sauser didn't know who wrote obituaries. Was it the funeral home? The deceased's loved ones? Mon, 22 Mar 2021 03:11:59 GMT High school football player in California collapses during game and dies A high school football player in Sacramento, California, has died after collapsing during a game. Mon, 22 Mar 2021 11:31:02 GMT Elon Musk responds to spying concerns in China: Trust us, we're just like TikTok Tesla CEO Elon Musk says his company's cars would never be used for spying in China, in response to reports that the country's military has banned the vehicles over such concerns. Mon, 22 Mar 2021 02:29:44 GMT A company will pay you $2,400 to stay away from screens for 24 hours Doom-scrolling getting you down? Mon, 22 Mar 2021 12:08:47 GMT Chris Columbus confirms there is an R-rated version of 'Mrs Doubtfire' -- but he has no plans to release it "Mrs. Doubtfire" director Chris Columbus has confirmed the existence of an R-rated version of the beloved 1993 family film. Tue, 16 Mar 2021 19:26:47 GMT See what delegate did after Marjorie Taylor Greene's gaffe CNN's Brianna Keilar speaks with Democratic Delegate of Guam Michael San Nicholas about bringing some of Guam's National Guard members to Rep. Marjorie Taylor Greene's (R-GA) office after she referred to Guam as a foreign country despite its status as a US territory during CPAC. Mon, 22 Mar 2021 09:07:32 GMT Is melatonin safe for kids? Here's what the experts say Melatonin is a popular over-the-counter sleeping aid, touted as a natural way for adults to fall asleep and used by millions around the world. So why wouldn't it be good for children, too? Fri, 19 Mar 2021 23:19:01 GMT Beloved sitcom actor reveals what led him to get sober 22 years ago Actor and Instagram sensation Leslie Jordan and Anderson Cooper discuss Jordan's decision to get sober 22 years ago. Watch "Full Circle" Monday, Tuesday and Friday at 6p E.T. (published missing) Your best refinance rates for March 2021 Sun, 21 Mar 2021 03:42:55 GMT A mother was killed in the spa shootings. A GoFundMe page for her sons has now raised millions A GoFundMe page for the two sons of Atlanta spa shooting victim Hyun Jung Grant had raised more than $2 million as of Saturday morning. Sun, 21 Mar 2021 22:36:42 GMT Connie Chung: The media is miserably late covering anti-Asian violence Media outlets are ramping up their coverage of anti-Asian violence following last week's Atlanta shootings, but many Asian-American journalists feel the issue has been ignored for far too long. Mon, 22 Mar 2021 06:14:22 GMT Hundreds gathered across the US to support Asian communities after Atlanta-area spa killings Nearly a week after eight people were killed at spas in the Atlanta area, hundreds gathered to remember the victims and call for an end to hate towards Asians in a year that has seen an uptick in attacks against members of the community. Sun, 21 Mar 2021 21:34:15 GMT Opinion: My parents came to the US from Korea for a better future. We must stop the hate My parents, like many immigrants, came to the United States from Korea for a better future. They were taught that this is the land of endless opportunities and that there were rules to follow in order to reach that better future. The rules were so strong, clear and always prevalent: to make sure they never took their eyes off the prize. These "rules" were supposed to keep the American ideal alive but in effect kept foreigners in their place and away from the majority. This is the "model minority" myth. And as is increasingly obvious, a myth is exactly what it is. Sat, 20 Mar 2021 22:06:19 GMT Spa owner killed two days before her birthday CNN's Natasha Chen speaks with the family of Xiaojie Tan, a spa owner who was killed during a string of shootings at Atlanta-area spas. Tan's daughter says she had plans to meet her mom the weekend before the shootings, but the meeting didn't happen and that was the last time they spoke. Sun, 21 Mar 2021 09:14:20 GMT She created a hate crime booklet for Asian Americans A Korean American woman living in Los Angeles has created a hate crime booklet for Asian American seniors to prepare them in case they are racially targeted. Sat, 20 Mar 2021 18:34:43 GMT Dad mics up his skiing toddler, captures her adorable pep talks Three-year-old Adia Leidums speeds down mountains on her tiny toddler skis in Fernie, British Columbia. Her parents Erich and Courtney say they are teaching Adia to ski at a young age in order to build confidence and resiliency. Mon, 22 Mar 2021 13:00:05 GMT Screen time and kids: What have we learned in the last year? My husband and I sent our 1st and 4th graders off to their last day of in-person school one year ago this month. I kissed them and pretended not to be terribly worried about the impending surge of patients to my hospital or the sacrifices we would all have to make to flatten the curve. Mon, 22 Mar 2021 12:06:34 GMT This startup aims to take the trash out of takeout Like many New Yorkers, Adam Farbiarz eats a lot of takeout. Sun, 21 Mar 2021 16:47:37 GMT A winery will pay you $10,000 a month to work and live rent-free in Sonoma If a post-work glass of wine is what you look forward to each day, maybe it's time to quit your day job. Wed, 17 Mar 2021 17:35:25 GMT See how a 6-year-old helped his little brother calm down A mother shared a video of her two sons on social media, where the older child is helping his younger brother calm down with breathing exercises. Sun, 21 Mar 2021 20:27:49 GMT Bella Thorne and longtime boyfriend Ben Mascolo are engaged Bella Thorne and her boyfriend, Italian singer Benjamin Mascolo, are engaged. Mon, 22 Mar 2021 12:30:16 GMT What's keeping America's top economists up at night There's a reason economics is frequently called the "dismal science." Sun, 21 Mar 2021 14:09:00 GMT Coming soon: Smart tires that could help you drive better We tend to think of tires as big circular hunks of black rubber that just roll down the road. We don't think about them much until something goes wrong and you're stuck waiting for roadside assistance or digging the spare out of the trunk. Sat, 20 Mar 2021 13:11:10 GMT Opinion: Why Austin is the next Silicon Valley More than a year ago now, I left the Bay Area for Austin, Texas. It wasn't an easy decision. I've spent three-plus decades embedded in Silicon Valley, investing in some of the region's top tech companies. With its entrepreneurial culture, top-tier academic institutions and countless tech success stories, there wasn't a better place in the world than the Bay Area to be an investor. (published missing) TripAdvisor's top 25 all-inclusive resorts for your post-Covid trip (published missing) Shop the top sleep sales happening this week before they're gone (published missing) We tried the Dodow sleep aid for over two months: It's worth it (published missing) Replace your old WFH set-up with these products we've tested (published missing) Good sleep starts with pillows: Here's an option for every sleeper (published missing) We spent weeks testing thermometers: These 3 were our top picks (published missing) 20 unique Amazon products you'll want to buy ASAP (published missing) Weighted blankets are all the rage: Here's what to know (published missing) Get Theragun-level relief for $200 cheaper with this massage gun (published missing) These $60 earbuds rival the AirPods in many ways (published missing) Get up to $200 off at restaurants with these Marriott credit cards (published missing) Rewards members: Score an extra 30% off with this Kohl's coupon Sat, 20 Mar 2021 15:31:45 GMT George Bush's moment of blinding clarity Former President George W. Bush did something unusual Thursday: he made news. He has spent his post-White House years perfecting his reclusive J.D. Salinger act, surfacing only to promote his books of portraits, and to join his odd-couple buddy Bill Clinton in supporting worthy causes. Mon, 22 Mar 2021 03:12:42 GMT The shameful vote by 172 Republicans On Wednesday, with the nation still reeling from news of the horrific shooting rampage in Atlanta that took the lives of eight people, seven of them women, the Democratic-controlled House of Representatives voted to reauthorize the Violence Against Woman Act (VAWA). Yet 172 House Republicans voted no on extending this landmark law, effectively choosing to protect a man's right to buy a gun over protecting the life of a woman. Sun, 21 Mar 2021 18:23:21 GMT How are the Atlanta spa shootings NOT a hate crime? At an office where I worked some years ago, a male co-worker would repeatedly stop by my desk and interrupt my work to talk about sex. One of my colleagues, who got fed up overhearing this, made a sexual harassment complaint on my behalf to the human resources department (without my permission, not that I cared). I got called in by HR and was told: "I have heard your complaint about racial discrimination. I think it is invalid." Sun, 21 Mar 2021 00:45:55 GMT Joe Biden's honeymoon is ending The honeymoon is about to end for President Joe Biden. He has sailed through his first two months in office, accomplishing a number of his goals with minimal backlash. But now the going is about to get tough and every new proposal is going to come up against Senator Mitch McConnell and the dreaded filibuster. Sun, 21 Mar 2021 13:24:35 GMT Who's teaching us to hate Where does the hate come from? That was the question on many people's minds after the killing of women at three spas in Georgia on Tuesday drew even more attention to the targeting of Asian Americans in the year since the coronavirus pandemic was declared. Sun, 21 Mar 2021 10:29:55 GMT Beulah Mae Donald's nightmare became a challenge to US In the evening hours of March 20, 1981, cold weather enveloped the city of Mobile, Alabama. The southern port was no stranger to the coolness of spring, but that night, Mobile struggled to climb out of 30-degree temperatures. Sat, 20 Mar 2021 16:38:15 GMT A country that treats you as a perpetual foreigner "You're a cute little tricksy." (published missing) Political Briefing: Voter suppression isn't new (published missing) Reliable Sources: A look at climate journalism (published missing) The Point: From Russia, with hostility (published missing) Politically Sound: Fmr. Secretary of State Madeleine Albright talks to CNN's Dana Bash (published missing) Axe Files: Atlanta Mayor Keisha Lance Bottoms (published missing) 2 minutes could save over $40K on your refi (published missing) 5 cards charging 0% interest until 2022 (published missing) Our 10 best stocks for 2021

It is fairly easy to get the RSS feeds working, updating, and saving.

It can be very time-consuming to figure out and handle quirks in how each site uses their RSS feeds in order to get useful information.

A common issue is character encoding issues.

Note: In this sample program, there were a number of character encoding issues from the CNN feed that were not resolved. Those are currently displayed as "(omitted - encoding issue)".

Many of these issues involve a site using extended ASCII encoding (sometimes with a code page) rather than, say, UTF-8 encoding.