Automatic pytest Testing and Failure Alert Monitoring
This is my solution to replace manually running pytest each day in command prompt. I want to automate running pytest every day, test if my automated python scripts ran smoothly and get notified if any tests fail.
Installing pytest, a Python testing library:
python -m pip install pytest
A Few Words on pytest
It is a unit test framework in python. pytest expects you to write each test as a self-contained function. One python file can contain many different test functions.
Writing a Test
Let's use test_file_date.py as our test, which uses the glob module and os.getmtime to get the csv with the most recent modification dateon my desktop. Then it tests if that date is today, in my case, for an expected daily file drop.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 | from datetime import datetime, date
import glob
import os
import getpass
def test_csv_date_equals_today():
"""The match format is for a Windows path with Unix style pattern matching."""
match = f"C:Users/{getpass.getuser()}/Desktop/*.csv"
# Get the most recent csv from a folder.
csv = sorted(glob.iglob(match), key=os.path.getmtime)[-1]
csv_timestamp = os.path.getmtime(csv)
csv_date = datetime.fromtimestamp(csv_timestamp)
print(csv_date.day)
print(date.today().day)
assert csv_date.day == date.today().day
|
Here's the pytest text output when the test is passing:
============================= test session starts =============================
platform win32 -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\
collected 1 item
..\..\Users\erick\Desktop\test_file_date.py . [ 14%]
[100%]
============================== 1 passed in 0.28s ==============================
Creating a Task with Windows Task Scheduler
If you haven't used python with Windows Task Scheduler before, my previous post on creating a task may help you. We'll create two of them for our testing system.
Adding Your Task Scheduler Program: a Windows .bat file
Add your username to the text below and adjust the paths to your system. Then save a Windows .bat file with this text, which points to your pytest.exe file:
cmd /c "C:\Users\your_username\Desktop\sandbox\Scripts\pytest.exe --capture=sys" ^
C:\Users\your_username\Desktop\test_file_date.py > C:\Users\your_username\Desktop\sandbox\Test_Results\Test_Results.txt
This example is referencing an .exe within a hypothetical "sandbox" virtual environment, located on my Desktop. If you have a virtualenv or venv, check the Scripts folder. (Bin on Linux.)
Breaking this out, there are five .bat files parts:
cmd /c "C:\Users\your_username\Desktop\sandbox\Scripts\pytest.exe --capture=sys"
Windows' cmd command takes a program, so we're passing pytest. The --capture=sys argument tells pytest to capture the test results. Note: switching cmd /c to cmd /k forces the terminal to stay open when you are testing your bat file. You can double-click your .bat file to test run it.
^
circumflex represents a line continuation in Windows batch files for better readability
C:\Users\your_username\Desktop\test_file_date.py
Next we're passing our python file as an argument to pytest, testing our file's modified date matches today's date.
>
This is a Windows redirect. It redirects the pytest output from sys to a text file, which is the last argument in our .bat file:
C:\Users\your_username\Desktop\sandbox\Test_Results\Test_Results.txt
Browse to select your .bat file for your Windows Task Scheduler task:
Reading the Tests and Triggering Alerts
Passing tests signal your scripts are running successfully. When things don't work, email alerts of the failure help us respond quickly.
Let's set another task scheduler job to run read_test_results.py, to run a few minutes after the first job each day. See this example of running Python with Task Scheduler if you haven't triggered a python script from Task Scheduler before.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31 | from datetime import date
import getpass
import logging
import os
"""Automate pytest with Windows Task Scheduler. Use Task Scheduler run a batch file.
The batch file runs pytest and captures our pytest function results to sys.
If a text file contains a failure or error, write the test contents into a folder.
"""
logging.basicConfig(
filename="Automated_Testing_Alerts.log",
level=logging.INFO,
format="%(asctime)s - %(message)s",
datefmt="%d-%b-%y %H:%M:%S",
)
logging.info("Checking for failed tests...")
directory = f"C:/Users/{getpass.getuser()}/Desktop/test_results/"
test_results = [fname for fname in os.listdir(directory) if ".txt" in fname]
for txt_file in test_results:
file_path = directory + txt_file
with open(file_path) as f:
text = f.read()
if "FAILURES" in text:
directory = f"C:/Users/{getpass.getuser()}/Desktop/send_failure_alert/"
name = f"{directory}{txt_file}_Failed_Results_{date.today()}.txt"
with open(name, "w+") as f:
f.write(name)
f.write(text)
else:
print("No failed tests found in file:")
print(txt_file)
|
Setting up Email Alert Notifications on a Box Folder
The above script checks the test results and creates a file with any failed tests in a different folder. I edited the folder's settings to email me when a new file is added, thanks to Box notifications. We use
Box for secure file sharing at my day current day job.
Alternatively for those without Box, you could use 'ole trusty smtplib to send the failure alerts with python. I chose the easier, ready to go option. Remember, "Simple is better than complex."
Also, the pywin32 module has an interface to Outlook that is very handy. For an example of sending a Windows Outlook email, check the very end of this post I wrote on "Scripting Windows Like a Pro".
We now have email monitoring alerts if our csv file date test fails, thanks to:
- Windows Task Scheduler (two tasks running each day in succession)
- python/pytest
- a Windows .bat file
- Box (or smtplib or pywin32) email alerts
In Summation
- The first task runs a .bat file to run pytest and create a text file with daily automation test results.
- The second task runs a python file. Setting in motion:
- Checking the test result text files, looking for failed tests
- Creating a text file with any failing tests in a Box folder, if found
- Then Box emails me if any test fails.
This was the first time I successfully deployed a Windows batch file. It took me many tries and googling to properly format of the .bat file. They are worth understanding and seem to open up many possibilities on Windows. In this case it was a "glue" piece that allowed me to accomplish my goal, automated testing and alerts for my python scripts.
What we learn along the way shapes us. Learning to use shell commands with Python scripts extends their abilities to help you get stuff done. I've benefitted from improving both my Windows and Ubuntu shell knowledge, which can sometimes be a handy substitute or complement to python. Now, time to write more tests. Thanks for reading!
See also:
pytest-csv: write test results to a csv with this plugin
Read more about software testing in my post here.
I am a beginner to testing my code. I wanted to write about testing to better understand it. While shaping this link fest masquerading as an essay, I collected great ideas from people who are way more experienced than me. You'll find a few of my thoughts, a Pytest example I use to monitor files, ideas for unit testing, property testing, test driven development and many other commonly used software tests.
An Introductory Rant on Testing
Over several years as a programmer, I've slowly grasped the landscape of testing in software development. After moving beyond my first few tutorials and projects, it seemed very noisy to sort out. Examples provided are usually simple assertions that seem tough to relate to a real use case. It might be easy test the wrong things. Plus, some don't do it at all! The quality of the tests is more important than the quantity. But what makes a quality test? Where's the balance between testing every minute detail of a program and not at all?
Yeah. And the worst thing that happens is that you get people that just stop thinking about what they’re doing. “This is the principle, to always write unit tests, so I’m always going to write unit tests,” and then they’re just not thinking about how they’re spending their time, and they wind up wasting a lot of it.
Joel Spolsky, Stack Overflow Podcast #38
Implementing software tests is a best practice for maintaining code, but seems ambiguous to someone who has not tested any code before. I guess the best way is
to read open source projects with test suites, but those can be tough to find. How do you know a good test suite when you see it? Maybe the maintainers went rogue and off the deep end with tests.
Online, everybody says you should test your code, is the emperor wearing any clothes?
As a beginner stumbling across articles on testing, these questions were tough to answer. As with most things in programming, figuring out the right question to ask is a challenge in itself. Codeacademy and Coursera never mentioned anything about writing tests. On the other hand, Django includes testing in its tutorial and documentation. Also, most languages come with built-in testing tools. Python has the unittest library.
Why test at all? First, some solid benefits of software testing:
- With tests on your code in place, you can implement changes and have confidence the code still works if the tests pass. This gives developers more confidence to iterate and improve an application.
- Detect problems faster. Passing tests are a good indicator that your programs are actually doing what they're supposed to do. If they don't pass, you likely found a bug you might have missed otherwise.
- When you find a bug, you either need to amend your code, or your tests. write a test for that bug and then fix it. Either that, or you need to be amend your tests. You've just improved the quality of your test suite.
- Automation. If you are writing tests, those tests can be automated. If you are manually checking the results of your program, you're missing a chance to automate those checks away. I haven't applied it yet, but have heard the Tox library may be useful to automate tests related to Python packaging. For more on automating tests, see this PyCon talk, Three Excellent Python Tools to Automate Repetitive Tasks.
- Test Driven Development can decrease the time spent debugging code. This claim sometimes lacks empirical evidence, supporting evidence tends to be anecdotal.
A good unit test, therefore, is one that helps enforce the contract to which the function is committed.
If a good unit test breaks, the contract is violated and should be either explicitly amended (by changing the documentation and tests), or fixed (by fixing the code and leaving the tests as is).
A good unit test is also strict. It does its best to ensure the output is valid. This helps it catch more bugs.
Pytest and Unit Testing in Python
This is where the Python hits the pavement. Unit tests are generally liked, although some prefer property tests or integration tests because they think the scope of unit tests is too narrow. The unittest library is Python's default testing framework. However nowadays, pytest seems to be the preferred unit testing framework for Python. Hypothesis is another popular framework I've read about.
Pytest Testing
Tests start to lose signal when Mock becomes routine instead of a reluctant workaround. - Brandon Rhodes, When Python Practices Go Wrong
Testing in Python \ General Unit Testing Ideas
I think hypothesis is probably underrated—some libraries are hesitant to incorporate it into their testing frameworks, but I think the property-based testing has real potential to catch scenarios humans would have a hard time anticipating, or at least that would take a long time to properly plan for. I find that hypothesis almost always adds a few useful test cases I hadn’t thought of that will require special error handling, for example.
Tyler Reddy, SciPy Release Manager
Integration \ Property Tests
Traditional, or example-based, testing specifies the behavior of your software by writing examples of it—each test sets up a single concrete scenario and asserts how the software should behave in that scenario. Property-based tests take these concrete scenarios and generalize them by focusing on which features of the scenario are essential and which are allowed to vary. This results in cleaner tests that better specify the software’s behavior—and that better uncover bugs missed by traditional testing.
Assertions
Assertions are generally accepted as welcome additions to your code.
In reality, the safety and restraints that these code carabiners provide actually give you more freedom to take risks in your coding. If you want to try out some risky feature, refactoring, or external library, you know something is wrong as soon as one of your assertions or tests fail and can undo back to an earlier working state.
Phillip J. Guo, Code Carabiners, (Link Broken)
Test Driven Development
Eventually, you'll discover the evangelists preaching Test Driven Development. There are certain discussions which polarize us in the software development world, such as the appropriate scenarios to deploy this system of development.
Opinions vary widely on the merits and appropriate application of TDD. I'm admittedly skeptical but do see the merits of TDD. But which flavor? Where do unit tests and integration tests fit in? How many tests should I write? What exactly should I be testing? This essay claims anyone pair programming software with an expected life of 3 or more years should use Test Driven Development.
"Test Driven Development is a tool for continuously evaluating hypotheses."
General Testing Ideas and Principles
Other common tests types:
Unit test: when it fails, it tells you what piece of your code needs to be fixed.
Integration test: when it fails, it tells you that the pieces of your application are not working together as expected.
Acceptance test: when it fails, it tells you that the application is not doing what the customer expects it to do.
Regression test: when it fails, it tells you that the application no longer behaves the way it used to.
Testing maturity level progression:
- No tests
- Occasional, slow, unreliable tests
- Semi-comprehensive integration tests
- Fast, comprehensive unit tests comprise the bulk of testing
- Dependency injection
- Composable subsystem design
- Real-time test feedback (ideally integrated into the editor)
- Tests are extremely reliable or guaranteed reliable by the type system
- With tooling that tracks the reliability of tests and provides that feedback to authors.
- Fuzzing, statistically automated microbenchmarking, rich testing frameworks for every language and every platform, and a company culture of writing the appropriate number of unit tests and high-value integration tests.
I recently wrote my first unit tests with pytest. Below is a script named test_file_date.py. It tests if the day of month of the most recently changed file in a directory matches today's day. To install pytest, enter into command prompt or terminal:
python -m pip install pytest
test_file_date.py
1
2
3
4
5
6
7
8
9
10
11
12
13 | import glob
import os
from datetime import datetime, date
# The dir_query format is for a Windows path with Unix style pattern matching.
def test_csv_date_equals_today():
dir_query = 'C:\\Users\\your_username\\Desktop\\*.csv' # specify csv extension and folder
file_path = sorted(glob.iglob(dir_query), key=os.path.getmtime)[-1] # get most recent file
file_timestamp = os.path.getmtime(file_path)
file_date = datetime.fromtimestamp(file_timestamp)
print(file_date.day)
print(date.today().day)
assert file_date.day == date.today().day
|
Run the test with pytest by entering:
pytest test_file_date.py
Conclusion
I write programs for personal productivity and to automate processes.
The scope of problems my code solves has grown with my programming ability. I'm now reaching the point where I can apply tests to my advantage.
However, sometimes I'll write a quick-hitter script for which I can't justify writing tests. Beyond those cases, testing can help if you pick
the right style for your project. More so for recurring, automated processes.
It feels pretty cool when your tests run and you know with more certainty whether a part of your program is getting the job done or not.
After setting up my first test with pytest, I have leveled up to novice tester, instead of blissfully not knowing what I don't know about testing. That's a step in the right direction.
GUI stands for "Graphical User Interface", aka the part of a program designed for human interaction. Adding a GUI to a Python
script allows anyone to run it without having to code or use the command line.
There are several GUI libraries in Python. A few I have heard of are Tkinter (comes in the standard library), wxPython, PyQT,
easygui, DearPyGui and PySimpleGUI. I explored Tkinter
back when I first got into Python. It was more intricate and offered more control over the look of your app, and took longer to pick up. Gooey is more of a pre-packaged GUI library.
The Gooey Github page was most useful to me and helped me to do what I needed.
The script posted in this blog helped as well. I needed to enable a human to
supply three files and enter a number. Gooey was a good match for this. The library has two branches:
- some basic widgets piggyback off the argparse library
- another part of the library uses a function called the GooeyParser. The GooeyParser offers more advanced widgets, like a file chooser. This was exactly what I was looking to use to pull in files for my script.
Installing Gooey
Argparse comes stock with Python. You can install Gooey via the pip installer. Open command prompt or terminal and enter:
python -m pip install Gooey
Below is a basic argparse/Gooey combination script. The argparse version offers a handful of widgets such as checkboxes and dropdown, but I had trouble getting them to work with the GooeyParser (used in 2nd script).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 | from argparse import ArgumentParser
from gooey import Gooey
@Gooey(program_name='Report Generator', default_size=(575, 600))
def get_args():
"""Demonstrating python's vars built-in to store arguments in a python dict."""
parser = ArgumentParser(description='A simple argument parser', epilog='This is where you might put example usage')
parser.add_argument('Name', action='store', required=True, help='Help text for option X')
parser.add_argument('Email', help='Help text for option Y', default=False)
parser.add_argument('Campaign Number', help='Help text for option Z', type=int)
parser.add_argument('Campaign Segment', choices=['A', 'B','All'], default='a', nargs='?')
user_inputs = vars(parser.parse_args())
print(user_inputs)
name = user_inputs['Name']
campaign_number = user_inputs['Campaign Number']
return parser.parse_args()
if __name__ == '__main__':
get_args()
|
Side note: Check out Python's vars() built-in function above.
It returns the input data as a dictionary called user_inputs. Then we can get the values via the dictionary's keys. Pretty nifty!
The @Gooey() part of the code is an advanced function known as a decorator in Python.
Put simply, decorators are functions that modify the function to which they are attached.
Below is my script that uses the more advanced GooeyParser for its "FileChooser" widget. Gooey allows you to group widgets together
and set how many widgets per line with the gooey_options={} parameter.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 | from gooey import Gooey, GooeyParser
@Gooey(program_name='Email Campaign Reporting Generator', default_size=(575, 600))
def get_args():
"""Adding two argument groups, each accepting two arguments. Using gooey_options to set layout."""
parser = GooeyParser(description='Export campaign report spreadsheets and upload below.')
top_group = parser.add_argument_group(gooey_options={'show_border': False,'columns': 1})
top_group.add_argument('Contact List', help='Upload Send List (.xlsx)', widget='FileChooser')
top_group.add_argument('Opens List', help='Upload Opens List (.xlsx)', widget='FileChooser')
top_group.add_argument('Unsubscribe List', help='Upload Unsubscribe List (.xlsx)', widget='FileChooser')
bottom_group = parser.add_argument_group(gooey_options={'show_border': False,'columns': 1, 'required':False})
bottom_group.add_argument('Campaign ID', action='store', help="Number found in the Campaign 'Reports' tab")
bottom_group.add_argument('Campaign Segment', action='store', help='Enter A, B, or All. All lists supplied must match segment.')
user_inputs = vars(parser.parse_args())
name = user_inputs['Name']
return parser.parse_args()
if __name__ == '__main__':
get_args()
|
Overall, Gooey knows what it wants to be, an easy to use GUI framework for Python. It does it well. Here's a screenshot of my program's shiny GUI:
Now that I have a GUI on top of my program and it delivers the expected output file, I'm hoping to take it one step further by packaging it up as a Windows .exe file.
This would allow it to run as a desktop app on any Windows computer without the need to install Python or library dependencies.
I've only begun exploring options to do this but a few libraries I've heard of are pyinstaller, cx_Freeze and Py2Exe. Updates coming if I figure it out. Cheers :D
Update: I did figure out how to compile my Gooey app to a Windows application with Pyinstaller. You can read more on how I did it here.