Browser automation using Python

Automated browser tests with python and Selenium

Maintaining a blog like this one requires different kinds of maintenance actions. For example removing outdated posts, updating posts and so on. One thing that I usually do is to test whether links still work and show the correct content.
Following the motto of Martin Fowler :

Now I’m a pretty lazy person and am prepared to work quite hard in order to avoid work.

I wanted to implement a way of doing that automatically. Therefore I created a python project that would search for specific links on my blog’s pages, tries to fetch the linked content and checks whether some keywords are still on it.
In this blog post I describe how I implemented that.

Additional goals

In addition to making this work, I also wanted to explore some other python / programming techniques:

• Test data should be defined in a json file for easy extension
• Evaluate pydantic for data types and validation
• Getting familiar wit GutHub Workflow and actions

Selenium setup

Selenium is a browser automation framework for writing functional test for web sites / application. It provides a generic API that can be bound to a wide range of browsers.
Setting up and using the Selenium WebDriver is easy to do in python; Install the package, install a driver and use it. The following code snippets show these steps. The complete code can be found in the test source code and the github workflow :

pip install selenium
cd bin
curl --output chromedriver.zip https://chromedriver.storage.googleapis.com/98.0.4758.102/chromedriver_linux64.zip
unzip chromedriver.zip

@classmethod
def create_web_driver(cls):
    options = Options()
    options.add_argument("--headless")
    current_path = pathlib.Path(__file__).parent.resolve()
    service = Service(executable_path=f"{current_path}/../bin/chromedriver")
    cls.driver = webdriver.Chrome(options=options, service=service)

In the python code, the web driver is created by specifying the executable path that points to the binary downloaded in the steps above.
Note that the option --headless is passed to the driver. This means that the browser will run without a GUI. This is particularly useful if run on a machine that does not have a display (e.g. CI/CD runners)

The test data model

Since the data model is actual quiet small, the word model might be a little over exaggerated. It basically consists of to entities; A blog post and a list of pages that are linked in the post. The model has been defined using pydantic and look like this:

from pydantic import BaseModel

class BlogPost(BaseModel):
    url: str
    pages: List[Page]


class Page(BaseModel):
    link_text_pattern: str
    keyword_patterns: List[str]

Deriving the model classes and defining their properties like this makes all properties mandatory and makes pydantic validate them when loading the model from json.
The json file for that model and the code loading the test data look like this:

[
  {
    "url": "https://tom1299.github.io/retro-blog/post/k8s-hardly-readable-configmap/",
    "pages": [
      {
        "link_text_pattern": "kubectl edit command",
        "keyword_patterns": [
          "kubectl edit",
          "KUBE_EDITOR"
        ]
  }
]

with open("test_data.json") as test_data_file:
    blog_post_test_data = json.load(test_data_file)
    for blog_post_data in blog_post_test_data:
        blog_post = BlogPost(**blog_post_data)
        blog_posts.append(blog_post)

In the code above, the json is loaded into a dict and then the ** operator is used to create the blog posts model (including the nested page objects).
The nice thing about the code is that all nested objects are created as well, the data is validated and the data classes do not need a constructor.
Using this approach I can easily add the tests just by extending the json test data file.

The test code

The code for doing the tests contains three steps:

• Get the blog post page
• Find the link(s)
• Get the linked content as text
• Verify that all keywords are in the content

Here are the code snippets for these tasks:

# Get the blog post page
web_driver.get(self.blog_post.url)

# Find the link
link = web_driver.find_element(By.LINK_TEXT, value=page.link_text)

# Click on the link and get the linked content as text
# click() might not work. See https://stackoverflow.com/a/52405269
web_driver.execute_script("arguments[0].click();", link)
page_text = web_driver.find_element(By.XPATH, value="/html/body").text

# Verify that all keywords are present
for keyword_pattern in page.keyword_patterns:
    match_found = re.search(keyword_pattern, page_text)
        if not match_found:
            raise KeywordNotFoundException(keyword_pattern, page.link_text)

There is one part in the code that is not really obvious: Instead of using the links click() method I use Javascript to click on it. The reason for that is that the link might (not yet) be visible on the page but is already in the HTML’s DOM . This might cause an ElementNotVisibleException. Instead of waiting for the element I opted for the slightly faster method of using JavaScript to click on the link.

The GitHub Workflow

Now to do the blog test automatically based on a schedule in GitHub, I created a workflow for that. The Following excerpt of the GitHub Workflow shows the steps to do that:

- name: Setup
    run: |
    mkdir bin
    cd bin
    sudo apt install -y curl unzip build-essential python3
    curl --output chromedriver.zip https://chromedriver.storage.googleapis.com/98.0.4758.102/chromedriver_linux64.zip
    unzip chromedriver.zip
    rm chromedriver.zip    
- name: PyTest
    run: | 
    pip install -r requirements.txt
    python -m pytest ./tests/*.py

The Setup step downloads and installs the driver and the PyTest step runs all tests using pytest. That’s it.

Conclusion

Selenium and it’ bindings offers a very elegant way for testing browser based applications or just plain websites. There are some challenges when doing these kind of tests like making these tests robust. Pydantic offers a nice and lean way of defining data models and populating objects from json. Combining these two allowed me to define tests by simply editing a json file. In conjunction with GitHub workflows I was able to created a scheduled test, notifying me if one of the linked pages is dead or does not contain the content I expected.
An alternative approach for doing that would be to use BDD instead of a json model like I did already for my prime numbers code kata .

Back to Home