Automatic pytest Testing and Failure Alert Monitoring
This is my solution to replace manually running pytest each day in command prompt. I want to automate running pytest every day, test if my automated python scripts ran smoothly and get notified if any tests fail.
Installing pytest, a Python testing library:
python -m pip install pytest
A Few Words on pytest
It is a unit test framework in python. pytest expects you to write each test as a self-contained function. One python file can contain many different test functions.
Writing a Test
Let's use test_file_date.py as our test, which uses the glob module and os.getmtime to get the csv with the most recent modification dateon my desktop. Then it tests if that date is today, in my case, for an expected daily file drop.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 | from datetime import datetime, date
import glob
import os
import getpass
def test_csv_date_equals_today():
"""The match format is for a Windows path with Unix style pattern matching."""
match = f"C:Users/{getpass.getuser()}/Desktop/*.csv"
# Get the most recent csv from a folder.
csv = sorted(glob.iglob(match), key=os.path.getmtime)[-1]
csv_timestamp = os.path.getmtime(csv)
csv_date = datetime.fromtimestamp(csv_timestamp)
print(csv_date.day)
print(date.today().day)
assert csv_date.day == date.today().day
|
Here's the pytest text output when the test is passing:
============================= test session starts =============================
platform win32 -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\
collected 1 item
..\..\Users\erick\Desktop\test_file_date.py . [ 14%]
[100%]
============================== 1 passed in 0.28s ==============================
Creating a Task with Windows Task Scheduler
If you haven't used python with Windows Task Scheduler before, my previous post on creating a task may help you. We'll create two of them for our testing system.
Adding Your Task Scheduler Program: a Windows .bat file
Add your username to the text below and adjust the paths to your system. Then save a Windows .bat file with this text, which points to your pytest.exe file:
cmd /c "C:\Users\your_username\Desktop\sandbox\Scripts\pytest.exe --capture=sys" ^
C:\Users\your_username\Desktop\test_file_date.py > C:\Users\your_username\Desktop\sandbox\Test_Results\Test_Results.txt
This example is referencing an .exe within a hypothetical "sandbox" virtual environment, located on my Desktop. If you have a virtualenv or venv, check the Scripts folder. (Bin on Linux.)
Breaking this out, there are five .bat files parts:
cmd /c "C:\Users\your_username\Desktop\sandbox\Scripts\pytest.exe --capture=sys"
Windows' cmd command takes a program, so we're passing pytest. The --capture=sys argument tells pytest to capture the test results. Note: switching cmd /c to cmd /k forces the terminal to stay open when you are testing your bat file. You can double-click your .bat file to test run it.
^
circumflex represents a line continuation in Windows batch files for better readability
C:\Users\your_username\Desktop\test_file_date.py
Next we're passing our python file as an argument to pytest, testing our file's modified date matches today's date.
>
This is a Windows redirect. It redirects the pytest output from sys to a text file, which is the last argument in our .bat file:
C:\Users\your_username\Desktop\sandbox\Test_Results\Test_Results.txt
Browse to select your .bat file for your Windows Task Scheduler task:
Reading the Tests and Triggering Alerts
Passing tests signal your scripts are running successfully. When things don't work, email alerts of the failure help us respond quickly.
Let's set another task scheduler job to run read_test_results.py, to run a few minutes after the first job each day. See this example of running Python with Task Scheduler if you haven't triggered a python script from Task Scheduler before.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31 | from datetime import date
import getpass
import logging
import os
"""Automate pytest with Windows Task Scheduler. Use Task Scheduler run a batch file.
The batch file runs pytest and captures our pytest function results to sys.
If a text file contains a failure or error, write the test contents into a folder.
"""
logging.basicConfig(
filename="Automated_Testing_Alerts.log",
level=logging.INFO,
format="%(asctime)s - %(message)s",
datefmt="%d-%b-%y %H:%M:%S",
)
logging.info("Checking for failed tests...")
directory = f"C:/Users/{getpass.getuser()}/Desktop/test_results/"
test_results = [fname for fname in os.listdir(directory) if ".txt" in fname]
for txt_file in test_results:
file_path = directory + txt_file
with open(file_path) as f:
text = f.read()
if "FAILURES" in text:
directory = f"C:/Users/{getpass.getuser()}/Desktop/send_failure_alert/"
name = f"{directory}{txt_file}_Failed_Results_{date.today()}.txt"
with open(name, "w+") as f:
f.write(name)
f.write(text)
else:
print("No failed tests found in file:")
print(txt_file)
|
Setting up Email Alert Notifications on a Box Folder
The above script checks the test results and creates a file with any failed tests in a different folder. I edited the folder's settings to email me when a new file is added, thanks to Box notifications. We use
Box for secure file sharing at my day current day job.
Alternatively for those without Box, you could use 'ole trusty smtplib to send the failure alerts with python. I chose the easier, ready to go option. Remember, "Simple is better than complex."
Also, the pywin32 module has an interface to Outlook that is very handy. For an example of sending a Windows Outlook email, check the very end of this post I wrote on "Scripting Windows Like a Pro".
We now have email monitoring alerts if our csv file date test fails, thanks to:
- Windows Task Scheduler (two tasks running each day in succession)
- python/pytest
- a Windows .bat file
- Box (or smtplib or pywin32) email alerts
In Summation
- The first task runs a .bat file to run pytest and create a text file with daily automation test results.
- The second task runs a python file. Setting in motion:
- Checking the test result text files, looking for failed tests
- Creating a text file with any failing tests in a Box folder, if found
- Then Box emails me if any test fails.
This was the first time I successfully deployed a Windows batch file. It took me many tries and googling to properly format of the .bat file. They are worth understanding and seem to open up many possibilities on Windows. In this case it was a "glue" piece that allowed me to accomplish my goal, automated testing and alerts for my python scripts.
What we learn along the way shapes us. Learning to use shell commands with Python scripts extends their abilities to help you get stuff done. I've benefitted from improving both my Windows and Ubuntu shell knowledge, which can sometimes be a handy substitute or complement to python. Now, time to write more tests. Thanks for reading!
See also:
pytest-csv: write test results to a csv with this plugin
Read more about software testing in my post here.
So you want to run your Python scripts automatically, but how?
I had heard of several popular scheduling libraries in Python like celery, Invoke, and schedule. One of my requirements is to run the python file "in the background", not in command prompt or an open window.
Enter Windows Task Scheduler, the de facto scheduler on Windows 7 computers. I have scheduled a few scripts and it is working like a charm. In this post, I will schedule an example script to clean up my desktop at the beginning of each day. I have a habit of accumulating many Excel files there throughout the workday. This example automatically moves them into a folder.
Other Windows scheduling alternatives worth mentioning include creating a Windows service, or using schtasks if you prefer the command line.
Here's how to schedule a Python script to run:
- Search for Windows Task Scheduler in the start menu. Then select "Task Scheduler Library" to see all of the tasks Windows is running automatically.
- In the right toolbar, select "Create Basic Task" and give it a name and description. Note: I selected "Configure for: Windows 7, Windows Server 2008 R2".
- Set the time and frequency that the program will run in the "Triggers" tab.
- Under the "Actions" tab, select "Start a Program" from the dropdown. Under "Program/Script", enter the path to your Python.exe file. I set mine to a Python executable located within my virtual environment, but yours might be found wherever you have Python installed.
C:\Users\your_username\Desktop\36env\Scripts\python.exe
- Under "Add arguments (optional)", add the path to your .py script, within quotes:
"C:\Users\your_username\Desktop\36env\clean_desktop_excels.py"
- Select additional conditions and settings as desired, such as "Wake the computer to run this task" and "Run with highest privileges".
I am enjoying this simple, easy and convenient scheduling manager for Windows. I figured most of this out thanks to this blog. Below is my script to clean my desktop each morning by moving my Excel files into a folder, using Python's stock shutil and os libraries. Set it and forget it, ya know what i mean? :D
1
2
3
4
5
6
7
8
9
10
11
12
13 | from shutil import move
import getpass
import os
# Desktop Spreadsheet File Cleaner: get all Desktop files and folders
src = f"C:/Users/{getpass.getuser()}/Desktop"
dir_items = os.listdir(src)
excel_files = [item for item in dir_items if ".csv" in item or ".xls" in item]
dst = f"C:/Users/{getpass.getuser()}/Desktop/Excels"
os.makedirs(dst, exist_ok=True)
for xl in excel_files:
path_to_file = src + xl
move(path_to_file, dst)
|
Additional Reading
Troubleshooting Windows Task Scheduler - Windows Documentation
