Sep 25, 2024
I began paying attention to what was relevant in the tech scene around 2014, in my 20s.
Back then, I was just getting started studying Python. It was an interesting time in tech.
The term "Big Data" was getting tossed around a lot, but the pandas library hadn't yet reached mass adoption
in data circles like it now has today. People were still talking about Hadoop + Map Reduce. (RIP)
In the 2010s, it didn't take much perusing online to find people in the Python community bashing Microsoft.
If tech companies were a high school, Apple was the cool kid everybody wanted to know, Microsoft was the
kid who nobody liked and everyone made fun of. Understandably, the Windows operating system didn't mesh
with Python programming as well as Linux or Mac OS. By 2024, Microsoft gained their mojo back, or found the mojo
they never had. Having used Windows a lot at my last job, I recognize the OS and its Python implementation have flaws.
I still got my work done and had no problems, without complaining. I continued to play around on Windows and write Python
on it even though people trashed it online. I'm glad I did!
How did Microsoft flip Apple? Steve Ballmer left the company in 2014, yielding to Satya Nadella as CEO.
Since then, the company culture shifted miraculously. In the Python community, they have made a huge impact
by investing in the language. They constantly release free Python + AI courses, and integrated Excel with Python.
Guido, the creator of Python is employed full-time, working on improving the Python language. That tells you a lot
of how much has changed since Python's BDFL is still working there after 3 years. Microsoft's culture change propelled it into
the 2020s with newfound momentum. With some timely bets, they saw the AI revolution coming and capitalized first.
If someone feels this way in 2024, they probably don't want to admit: Microsoft is Apple in 2012,
and Apple is Microsoft in 2012.
What is funny to see is that nowadays, fewer people are bashing Microsoft. I used to see it regularly,
people teeing off online, "writing Python on Windows is such a terrible experience for XYZ thing, why is Windows so awful??""
I see less of those people posting such thoughts now. Maybe they're still out there. If someone feels this way in 2024,
they probably don't want to admit: Microsoft is Apple in 2012, and Apple is Microsoft in 2012. I posit they
switched places in respective coolness among tech circles. People realized Apple is not the friend of developers
or society in general. They are self-serving to a vicious degree. Apple is focused on maintaining their walled garden on iOS.
Microsoft is now a better advocate for techies and Python development. Sure, some people prefer to code on Macs,
more power to them. Linux is typically the favorite of the three and it is awesome. It's also not released by a
for profit corporation which is uber cool to developers.
Apple is also less cool due to their battle with Epic Games and insistence on 30% rake for in-app
purchases on iOS. Not to mention an unwillingness to change their policies to appease stricter European Union regulations
for things like 3rd party app stores.
Microsoft is integrating AI deep into their products. Apple, after being slow on the uptake to AI,
followed Microsoft's lead to invest in OpenAI and roll its AI chat to iPhones.
Who is the leader here? In terms of "What have you done for me lately?", it's Microsoft. In terms of who supports open
and free information, it's Microsoft. Who's cool now?
Sep 07, 2024
Bots and robots.txt
Bots are crawling the internet. They make up a large chunk of all web traffic.
On my blogs, I've always used a robots.txt
to say to bots, "scrape away!" like this:
User-agent: *
Disallow:
Some websites use a robots.txt to say, "don't scrape our content".
We can ask crawlers to restrict from crawling, but this is more or less an honor system.
For this blog, I always welcomed all real traffic but activated Cloudflare's "Bot Fight Mode" to
to protect against some of the bots.
Cloudflare Rolls Out New AI Blocker Configuration
I noticed in my Cloudflare Pages dashboard that a new setting has become available to me on the free plan.
Block AI Bots Cloudflare Pages ConfigurationThis new configuration option gives another level of security if I want to use it.
For now, I'm choosing to keep my blogs open to most traffic, including AIs while excluding some bots.
Anyone who publishes to the internet needs to assess their willingness to trust big tech's bots to scrape their data.
You can block artificial intelligence (AI) bots, crawlers, and scrapers from scraping
your website content and training large language models (LLM) to recreate it without
your permission. When you enable this feature via a pre-configured managed rule,
Cloudflare can detect and block AI bots from your website.
- AI Bots
If an LLM is just a remix of all the data it consumes, how do we trace the origin of its results?
And how does that get attributed back? It seemed easier to tell with Google's link based search results.
But even then, there was a lack of transparency. It's still that way today. I appreciate that Cloudflare is putting
this choice to block or not block AIs in the hands of website makers. They're also releasing
an "AI Audit" tool that lets website owners set a price for their sites with AI crawlers.
This murkiness leaves site owners with a hard decision to make.
The value exchange is unclear. And site owners are at a disadvantage while they play catch up.
Many sites allowed these AI crawlers to scan their content because these crawlers,
for the most part, looked like "good" bots - only for the result to mean less traffic
to their site as their content is repackaged in AI-written answers
...
We think that sites of any size should be fairly compensated for the use of their content.
Cloudflare plans to launch a new component of our dashboard that goes beyond just blocking
and analyzing crawls. Site owners will have the ability to set a price for their site,
or sections of their site, and to then charge model providers based on their scans
and the price you have set.
- Start auditing and controlling the AI models accessing your content, Cloudflare Blog
Crawler Traffic on lofipython.com, Aug. - Sept. 2024The point of this post is to reflect on the nuances of web creators and their relationship with big tech.
Everyone will have to decide for themselves if they think it's worth it. In this case, we're likely "paid in exposure".
Big tech will distill the value of what we write to inquiring humans and reference back with links. Will website creators
lose out on this exchange? Are we cutting off a channel of distribution if we push back against LLMs scooping up our data?
Right now there seems to be more questions than answers. We'll see.
Disclosure: the author is a Cloudflare investor, going on 3 years holding shares.
Aug 29, 2024
Ubuntu allows aliasing commands to run a shell function. Below, I used
xdg-open to open my Cloudflare Pages
dashboard with an alias.
Add Ubuntu Function to .bashrc
I used VS Code to add this to my .bashrc file. Now, when I type cloudflare
into my shell it launches the dashboard in Chrome. Remember to close and reopen
a new shell before testing out the command.
| open_cloudflare() {
xdg-open "https://dash.cloudflare.com"
}
alias cloudflare=open_cloudflare
|
The pattern of being able to open a page in a web browser with a quick command could
be applied to lots of my frequently visited websites.
Bonus Alternate Version: Python webbrowser Module CLI
Sometimes the right tool is Python. Other times, the Linux shell CLI tools are sufficient.
Since this is a Python blog... here is a version that leverages the
Python webbrowser module CLI that
also works, assuming you're already in your Python environment.
| open_cloudflare() {
python -m webbrowser https://dash.cloudflare.com
}
alias cloudflare=open_cloudflare
|
Aug 12, 2024
I've been messing around with Apps Script lately
and am finding it useful. The below Apps Script was created with a few prompts to Bing Copilot. First, I asked Bing:
Write a Google sheets formula that copies the value in cell B10 to another sheet and also fills another column with today's date
...and a later prompt:
how can i make this script add a new row each time in the target sheet and appending to existing rows?
I refined with some followup prompts to write a script. It copies cell B10's value
from a sheet then fills in a row in a different sheet with cell B10's value and today's date.
AI allows me to do things that would have taken more time to figure out before. For example,
Bing just tapped the appendRow
function to handle the updating of values without overwriting existing data. We don't need to know the specific
Apps Script functions to write code in a new language anymore. AI can just fill in those details for us.
Apps Script to Copy a Cell Value and Create New Row with Today's Date
In this example, one sheet is named "Summary" and the other target sheet is named "Daily"
with 2 column names.
1
2
3
4
5
6
7
8
9
10
11
12
13
14 | function copyTotalValue() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sourceSheet = ss.getSheetByName("Summary");
var targetSheet = ss.getSheetByName("Daily");
// Get the value from B10 in Summary
var value = sourceSheet.getRange("B10").getValue();
// Get today's date
var today = new Date();
// Append a new row to the Daily sheet with the value and today's date
targetSheet.appendRow([value, today]);
}
|
According to Bing, Apps Script is a JavaScript dialect:
Google Apps Script is based on JavaScript. It uses a subset of JavaScript and provides additional built-in functions...
Automate The Script From the Apps Script Jobs Dashboard
Go to the Apps Script Jobs dashboard.
Add a new script to run Apps Script.
Click the blue "Add Trigger" button to create a new job.
Supplementary Reading
Apps Script Reference Overview
Apps Script API
Apps Script Spreadsheet Service
Jul 25, 2024
This post focuses on Google Sheets formulas, rather than Python. If you're interested in accessing Google Sheets with Python,
check out this post I wrote about the pygsheets library. I haven't personally tried it, but
xlwings also looks like a decent option if you're looking for Google Sheets Python libraries.
Here are some functions I recently discovered to analyze data in Sheets. There a few different strategies you can use to import data into Google Sheets with the functions available.
Below, you'll see some different approaches you use to get data into Google Sheets and then analyze it. The Google Sheets formulas rabbit hole is deep.
Lots of power can be harnessed by getting familiar with the formulas it has built-in.
Year To Date SPARKLINE of a Stock
=SPARKLINE(GOOGLEFINANCE("Nasdaq:NVDA", "price", DATE(YEAR(TODAY()), 1, 1), TODAY(), "daily"), {"charttype", "line"; "linewidth", 2; "color", "green"})
Year to Date SPARKLINE of USD to MXN Currency Value with GOOGLEFINANCE + SPARKLINE USING IF to Dynamically Color the SPARKLINE
=SPARKLINE(GOOGLEFINANCE("CURRENCY:USDMXN", "price", TODAY()-365, TODAY(), "DAILY"), {"charttype", "line"; "linewidth", 2; "color", if(P22>0,"green","red")})
Note the use of an if condition to check another cell and set the color to green if > 0 or red if < 0.
Market Cap with GOOOGLEFINANCE
=GOOGLEFINANCE("Nasdaq:NVDA", "marketcap")
Price/Earnings Ratio
=GOOGLEFINANCE("Nasdaq:MSFT", "pe")
Daily % Change of a Stock
=GOOGLEFINANCE("Nasdaq:TSLA","changepct") &"%"
Import the Price of a Cryptocurrency with IMPORTDATA
=IMPORTDATA("https://cryptoprices.cc/ADA")
This function imports the price of Cardano cryptocurrency from cryptoprices.cc.
Import the Market Cap of a Cryptocurrency with IMPORTDATA
=IMPORTDATA("https://cryptoprices.cc/ADA/MCAP")
This function imports the current market cap of Cardano cryptocurrency.
Import the Daily % Change of Ethereum Cryptocurrency with IMPORTXML and INDEX
=TEXT(
IF(
IMPORTXML("https://coinmarketcap.com/currencies/ethereum/", "//p/@data-change") = "down",
"-" & INDEX(IMPORTXML("https://coinmarketcap.com/currencies/ethereum/", "//p[@data-change]"), 1, 2),
INDEX(IMPORTXML("https://coinmarketcap.com/currencies/ethereum/", "//p[@data-change]"), 1, 2)
),
"0.0%"
)
This method uses IMPORTXML to import data to google sheets by passing an "xpath query". The first line checks if the direction of the % change is "down". If it is down,
then we know the % change is negative, otherwise the % change is positive. In this case I inspected the source code of coinmarketcap to reference the class names of the HTML I was targeting.
Conditionally sum a range with SUMIFS, SUMIF and COUNTIF
=SUMIFS('sheet_name'!G:G, 'sheet_name'!N:N, ">0")
In the following examples, "sheet_name" = Your Google sheet's name. This function sums the corresponding cells in column N if column G contains a number greater than 0.
=SUMIF('sheet_name'!N:N, "Some Value",'sheet_name'!G:G)
This function sums all values in column G if the values in column N are equal to "Some Value".
=COUNTIF('sheet_name'!A:A, "Some Value")
Count all the cells in column A that equal "Some Value".
Select Columns from a Dataset
{ 'sheet_name'!A:D, 'sheet_name'!T:X, 'sheet_name'!Z:AA}
Google sheets recognizes this format of bracket enclosed ranges of columns to select into your dataset. In the next example, you can see this applied.
Conditionally Select a Range of Cells from a Dataset with SORTN, FILTER, and REGEXMATCH
=SORTN(FILTER({'sheet_name'!A:A, 'sheet_name'!E:E, 'sheet_name'!P:P/1,'sheet_name'!F:F}, REGEXMATCH('sheet_name'!P:P, "-")), 5, 0, 3, TRUE)
This formula constructs a dataset, then filters it on a condition using REGEXMATCH to check if the cell contains a hyphen (-).
5 specifies how many rows to return, and 3 specifies which column to sort on, in this case the 3rd column.
Conditionally Select a Range of Cells from a Dataset with SORTN and QUERY with SQL-Like Language
=SORTN(QUERY({'sheet_name'!A:A, 'sheet_name'!E:E, 'sheet_name'!P:P,'sheet_name'!F:F}, "SELECT * WHERE Col3 IS NOT NULL"), 6, 0, 3, FALSE)
This queries rows that are not null, containing data.
Official Function Documentation
Google Sheets Function List
GOOGLEFINANCE
IMPORTDATA
IMPORTXML
SPARKLINE
FILTER
SUMIFS
QUERY
SORTN
INDEX
REGEXMATCH
Jun 25, 2024
While walking down a busy avenue in Mexico, I listened to a conversation in Spanish. I'm not fluent in Spanish,
but I have 4 years of high school classes and ~2 more years of real world experience. Spanish is not spoke in a bubble.
It needs to be practiced to be learned. I put much effort into listening. There is much more I have to learn. Typically,
I try to compensate for some lack of skills with intense focus. Sometimes I know what is said and more often
I am closer to guessing.
On this walk down the street, I attempted to listen to my friends speak. In the back of my mind, I was frustrated
because the noise of the cars going past us and a city in motion made it near impossible to hear the words. Similarly, attempting
to listen in a bar, party or anywhere playing music is very tough.
I used to think, "if only it was more quiet!" while walking and straining to hear what my friends said over the noise.
Today I had an epiphany: there's always background noise.
A difference between textbooks and real life: there's usually noise in the background.
We should embrace practicing and growing our skills because there will always be "noise".
This is a tidy metaphor for distraction and our focus bandwidth. In the lens of coding, the "noise" might be
skill deficiencies and lack of understanding, other co-workers, unimportant projects, meetings, social media, games
and anything that misdirects us away from our primary goals.
It's easy to get annoyed when there is excessive noise. It makes it way harder to hear what is being said.
Nonetheless, I don't have to let the noise distract my effort and intention to learn. So it is with programming and all skills.
There will always be "noise". Embrace the noise and lean heavily on your intention to practice.
Image Generated with Bing Image Creator
May 01, 2024
You can easily share a Google Calendar event if you know the url syntax Google uses.
When the url is opened in a browser, it prompts the person you want to share an event
with to save it to their calendar.
Demonstrating Google Calendar URL Arguments in Python
By simply knowing the proper url arguments, you can enable people to quickly save a Google Calendar event.
This example uses the Python standard library: urllib to format the Google calendar url parameters and webbrowser
to open the url in a browser. This is a handy little trick to keep in your back pocket!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38 | from urllib.parse import urlencode
import webbrowser
def new_google_calendar_event():
"""
Pass an event to Google Calendar with url arguments.
Base URL: https://calendar.google.com/calendar/render
URL Arguments:
action: TEMPLATE
text: Event Title
dates: start_datetime/end_datetime
details: event description or link to more info
location: url to webcast, call or physical location name
ctz: set the time zone by name, ex: America/New_York
recur: set a recurring event, ex: RRULE:FREQ%3DWEEKLY;INTERVAL%3D3
crm: if Free, Busy, or Out of Office, ex: AVAILABLE, BUSY or BLOCKING
add: add a list of guests by email, ex: [email protected],[email protected]
"""
parameters = {
"action": "TEMPLATE",
"text": "Title of Event",
"dates": "20240504T123000Z/20240504T133000Z",
"details": "Event Details: https://lofipython.com",
"location": "link to webcast, call or physical location",
"ctz": "America/Chicago",
"crm": "BUSY"
}
# Returns str of URL encoded parameters.
url_parameters = urlencode(parameters)
url = f"https://calendar.google.com/calendar/render?{url_parameters}"
print(url)
return url
url = new_google_calendar_event()
webbrowser.open_new(url)
|
I was struggling to find any official documentation, so I figured Google's Gemini AI model might know where this is documented.
Using the app on my phone, Gemini informed me of a useful Google Calendar Help thread response from Neil@GCalTools.
- The official documentation says to use "https://calendar.google.com/calendar/render" instead of "https://calendar.google.com/calendar/event", but they both work, at least for now.
- - Neil@GCalTools, Google Calendar Help
Relevant Links
Read more documentation of possible url arguments in the add-event-to-calendar-docs Github repo.
Wikipedia Time Zone List
Recurrence Rule Syntax
Kalinka Google Calendar Link Generator
Apr 07, 2024
Back in 2016, I built a web2py app
as my first web application. It accepts a prompt from the user and
primitively attempts to match that text to a line of Kevin Parker's lyrics written for the band Tame Impala.
I didn't look at the app's code for many years. I just needed to log into PythonAnywhere
every 3 months and hit a button to keep it running on their free plan. Until I decided to update it recently.
Tame Impala released a new album and lots of additional tracks to import to the app.
Just need to update the database with new music. No big deal, right?
Part of the problem with updating the app was that I didn't remember where the important code
like the controller default.py and relevant HTML files were after not seeing it for 5+ years.
It took some time to remember the folder structure of a web2py project. Since it was my first
project ever, documentation was nonexistent. I could have saved myself a lot of grief if I'd wrote
down some notes when I made the app.
web2py is relatively easy to grasp for Python developers. One thing I like is that
once it is installed, the development server is easily started by running the web2py.py file:
cd web2py
python3.10 web2py.py
web2py Python Errors Solved
I installed web2py locally with the help of a DigitalOcean blog post.
After I failed to push a new version of the app to production, for some reasons it was in a broken state.
Python version issues surfaced, requiring some heady navigation. Enjoy these gritty details
of the tracebacks that transpired.
ModuleNotFoundError: No module named 'formatter'
This error showed up in my app's WSGI error logs. Initially, I researched and attempted to install
the formatter module. I believe this was caused by attempting
to run Python code compiled to a .w2p file on Python 3.11 on a Python 3.10 interpreter. However, I didn't
know how to solve it until after I saw the next error.
SystemError: compiled code is incompatible
After reading this error, I consulted Bing about it. One of the options that Bing suggested was
that my Python code had incompatible versions. This was caused by a mismatch between my development
and production Python versions.
Installing Python 3.10 in Development Environment
Originally, I compiled the updated web2py app in Python 3.11 on my Chromebook. My PythonAnywhere environment was
running Python 3.10. Therefore, I need to build the
development code in Python 3.10 to match the production environment on PythonAnywhere.
I entered a handful of commands from Bing Copilot to build Python 3.10 on my Ubuntu development environment:
sudo apt-get install build-essential
sudo apt-get install zlib1g-dev
sudo apt-get install libsqlite3-dev
wget https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tgz
tar zxvf Python-3.10.5.tgz
cd Python-3.10.5
./configure --enable-optimizations --enable-loadable-sqlite-extensions
make && sudo make install
The lesson I took away is to consider your production environment's Python version before you begin working on a project.
In most cases, you'll want to match that version in your development environment to avoid errors like this.
After compiling the new development Python 3.10 version, I exported the app to a new .w2p file.
Next, I imported the .w2p file containing the updated app to PythonAnywhere in the admin interface app importer.
After syncing my development and production environment versions, the app showed a different error.
Progress!
Since I was using a .w2p file from 5+ years ago, it contained old Python web2py code written in
earlier Python versions with a few more bugs. Despite these version inconveniences, I was happy to see the
"compiled code is incompatible" and "formatter module missing" errors stopped.
One problem solved, two more discovered in its wake, am I right?
SyntaxError: multiple exception types must be parenthesized
This error showed up in my appadmin.py. At some point this unparenthesized syntax was phased out of Python.
The fix is add parentheses to the exception statements:
Incorrect
Correct
unable to parse csv file: iterator should return strings, not bytes (the file should be opened in text mode)
In order to import the new Tame Impala songs to the SQLlite database, web2py provides a
GUI interface in its admin panel or the DAL (Database Abstraction Layer).
I chose to use the GUI. In the GUI, you can either manually enter each song or use its csv import widget.
To save time, I imported via the csv widget. However, this error slowed me down.
It stemmed from the need for TextIOWrapper to convert the csv data to a required format.
The solution I found was to use the
fix suggested by AnooshaAviligonda.
In web2py/gluon/packages/dal/pydal/objects.py, I swapped in this code:
| csv_reader = csv.reader(TextIOWrapper(utf8_data,encoding), dialect=dialect, **kwargs)
|
After adding the above code to my web2py app's objects.py file, the csv importer completed my
new Tame Impala songs database import. Also, I was able to export an app from my development environment
and deploy it into PythonAnywhere via the admin interface. Mission accomplished.
I imported the new songs to my app and brought the code forward into future Python versions.
Keeping up with this project over the years shows how maintaining an app across different Python versions
can cause unexpected challenges. With these Python tracebacks conquered, the app is back on the web.
Now with all of Tame Impala's new lyrics!
Check out my Tame Impala web2py app here:
tameimpala.pythonanywhere.com/tameimpala
Mar 31, 2024
This post shows how to set up a custom Django template tag filter. With help from Django's load built-in,
you can execute a Python function from your app's HTML. In this example, the function returns the quotient,
or result of dividing two numbers. Mathematics and Python for the win!
I worked some of this out with help from Bing and
following along with the Django custom template tags documentation.
Install Django Python Library
pip install Django
Create templatetags.py
I created a "templatetags" folder in the app folder and placed templatetags.py within it.
The Django docs recommend your app's folder. Below, the @register.filter decorator registers the divide function so Django knows it exists.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 | from django import template
register = template.Library()
@register.filter
def divide(value, divisor):
"""A Django filter that accepts 2 arguments:
1. value, number to be divided
2. divisor, number to divide by
Returns the quotient in hundredths decimal format.
"""
# Check if the argument is zero to avoid division by zero error.
if divisor == 0:
return None
quotient = value / divisor
return f"{quotient:.2f}"
|
Edit Your HTML Code to Call the Divide Function
1
2
3
4
5
6
7
8
9
10
11
12
13 | {% extends 'base.html' %}
{% block content %}
{% load templatetags %}
<div class="container" style="display: inline-block; inline-size: 90%; block-size: auto; writing-mode: horizontal-tb;">
{% for hotel in hotels %}
<div class="row row-cols-3" style="display: flex; justify-content: flex-end; padding: 10px; margin: 10px; background-color: #f0ffff; box-shadow: 5px 5px 10px gray; border-radius: 10px;">
<div class="col">
<h5 style="color: green;">Peso: ${{ hotel.price }}</h5>
<h5 style="color: gray;">USD: ${{ hotel.price|divide:18.9 }}</h5>
</div></div>
{% endfor %}
</div>
{% endblock %}
|
In the HTML, call the divide function by loading the templatetags module and then passing two numeric arguments:
{{ hotel.price|divide:18.9 }}
- A number to be divided, here the hotel price from a "Hotel" DB model
- The function name to call and a number to divide by. Here we use "divide:18.9" to approximately convert Mexican pesos to US dollars. Recently, the exchange rate fluctuates in the 17-19 pesos per dollar range.
Basic Hotel Model Example
| from django.db import models
class Hotel(models.Model):
name = models.CharField(max_length=200)
price = models.DecimalField(max_digits=10, decimal_places=2)
address = models.CharField(max_length=200)
city = models.CharField(max_length=200)
all_inclusive = models.BooleanField()
photo = models.ImageField(upload_to="media")
|
Register Model in admin.py
| from django.contrib import admin
from .models import Hotel
admin.site.register(Hotel)
|
I was pleased to be able to make some on the fly mathematic calculations in my HTML
with a custom Django filter!
Mar 01, 2024
The posts on this blog are written in reStructuredText.
I recently had an idea to write a Python script to check my .rst document url links for broken tags or urls that are not valid.
When I'm using Pelican, it hints when a url tag is malformed and gives a line number.
However, it can still be difficult to track down a problematic link when there's an issue in your document.
So I did what any other programmer probably does in 2024, opened up an AI assistant. Bing quickly generated the structure
of the Python script from my prompt, but like usual it required some tinkering and refinement to make it work.
You can find the complete Python script in the rst-url-validator Github repo.
Bing was also valuable for helping me modify the regex module code
for rst-url-validator. It is typically very confusing to reason about regex,
but with AI assistance I can just ask it for regex that does something and
the AI can generate the code. Coding is getting way easier than it used to be thanks
to these new advances in large language models.
Install Python Library Dependencies
The script uses requests to send an HTTP request to the url
and rich to print color-coded output to the terminal screen.
pip install requests
pip install rich
Clone The Github Repo
git clone git@github.com:erickbytes/rst-url-validator.git
cd rst-url-validator
Run The CLI Command
python rst-url-validator.py your_file.rst
It parses each url tag, checks for required characters and sends a HEAD request
to the url to check if it loads successfully. Some websites return status codes like 403 (permission denied), 301 (redirect), or 404 (page not found).
If the status code returned is 200, it's safe to say the page loaded. Be aware that some sites like Cloudflare (returns 403) and Amazon (returns 503) do not serve
any requests and just return an error. In cases like these, you'll need to check pages like this manually in a browser to see if the page loads.
This script helped me find and fix or remove a bunch of broken links on the older posts on this blog. If you're working with .rst documents, give it a try!