Python Automation

Automation refers to the process of using scripts or programs to perform tasks automatically without manual intervention. With Python, you can automate all kinds of tasks, from file operations and system processes to web scraping and browser automation.

In this lesson, we’ll explore how to improve YOUR efficiency by automating repetitive tasks with Python libraries like os, shutil, and Selenium.

Why Use Python for Automation?

• Ease of Use: Python’s simple syntax makes it easy to write scripts for automating tasks.
• Extensive Libraries: Python has libraries for interacting with the file system, web browsers, databases, and more.
• Cross-Platform: Python scripts can be run on Windows, macOS, and Linux, making it versatile for automating tasks across different environments.

Automating File and Directory Operations with os and shutil

Python’s os and shutil modules allow you to interact with the file system, making it easy to automate file and directory operations such as creating, deleting, and moving files.

Working with Files and Directories Using os

The os module provides functions for interacting with the operating system, including handling files and directories.

Example: Creating a directory and listing its contents.

import os

# Create a new directory
os.makedirs('my_directory', exist_ok=True)

# List contents of the current directory
print(os.listdir('.'))

• os.makedirs(): Creates a new directory. The exist_ok=True argument ensures no error is raised if the directory already exists.
• os.listdir(): Lists all files and directories in the specified path ('.' represents the current directory).

Copying, Moving, and Deleting Files with shutil

The shutil module provides high-level file operations such as copying, moving, and deleting files.

Example: Copying and moving files.

import shutil

# Copy a file from source to destination
shutil.copy('source.txt', 'destination.txt')

# Move a file
shutil.move('source.txt', 'new_folder/source.txt')

# Delete a file
os.remove('destination.txt')

• shutil.copy(): Copies a file to a new location.
• shutil.move(): Moves (or renames) a file.
• os.remove(): Deletes a file.

Walking Through Directory Trees

You can use os.walk() to recursively walk through a directory tree, processing files and subdirectories.

Example: Printing all files in a directory and its subdirectories.

for dirpath, dirnames, filenames in os.walk('my_directory'):
    print(f'Found directory: {dirpath}')
    for filename in filenames:
        print(f'File: {filename}')

Automating System Tasks with os

You can use the os module to automate system-level tasks, such as executing system commands, retrieving environment variables, or managing processes.

Running System Commands

You can run system commands directly from Python using os.system() or the more powerful subprocess module.

Example: Running a system command.

import os

# Run a system command (e.g., list directory contents)
os.system('ls' if os.name != 'nt' else 'dir')

• os.system(): Runs a system command. The example runs the ls command on Unix-based systems or dir on Windows to list directory contents.

Managing Environment Variables

You can read and set environment variables using the os.environ dictionary.

Example: Accessing an environment variable.

import os

# Get the value of the HOME environment variable
home_dir = os.environ.get('HOME', 'Not set')
print(f'Home directory: {home_dir}')

• os.environ.get(): Retrieves the value of an environment variable (HOME in this case). If the variable is not set, a default value ('Not set') is returned.

Automating Web Browsers Using Selenium

Selenium is a powerful tool for automating web browsers. You can use it to perform tasks such as interacting with web pages, filling out forms, and navigating between pages. Selenium is commonly used for web scraping and automating web-based workflows.

Installing Selenium

To get started with Selenium, you need to install the Selenium library and download a web driver (e.g., ChromeDriver for automating Chrome).

pip install selenium

You’ll also need to download the appropriate web driver for your browser (e.g., ChromeDriver, GeckoDriver for Firefox). Place the driver in a location accessible by your Python script.

Basic Example: Automating a Browser with Selenium

from selenium import webdriver

# Set up the Selenium WebDriver (e.g., Chrome)
driver = webdriver.Chrome(executable_path='/path/to/chromedriver')

# Open a website
driver.get('https://www.example.com')

# Find an element by its tag name and print its text
heading = driver.find_element_by_tag_name('h1')
print(heading.text)

# Close the browser
driver.quit()

Filling Out Forms and Submitting Data

You can use Selenium to interact with web forms by finding form elements and sending input to them.

Example: Filling out a form and submitting it.

# Open a webpage with a form
driver.get('https://www.example.com/login')

# Find the username and password fields
username_field = driver.find_element_by_name('username')
password_field = driver.find_element_by_name('password')

# Enter data into the fields
username_field.send_keys('my_username')
password_field.send_keys('my_password')

# Find the submit button and click it
submit_button = driver.find_element_by_name('submit')
submit_button.click()

Waiting for Elements to Load

Web pages often load elements dynamically, so Selenium provides waits to handle these situations. The WebDriverWait class allows you to wait until a specific condition is met.

Example: Waiting for an element to be clickable before interacting with it.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Wait until the element is clickable
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.NAME, 'submit')))

# Click the element
element.click()

Automating Email Tasks

Python’s smtplib module can be used to automate sending emails. This is useful for notifications, reports, or any task that requires sending automated messages.

Example: Sending an Email Using smtplib

import smtplib
from email.mime.text import MIMEText

# Email details
sender = 'your_email@example.com'
recipient = 'recipient_email@example.com'
subject = 'Automated Email'
body = 'This is an email sent by Python.'

# Create the email message
msg = MIMEText(body)
msg['Subject'] = subject
msg['From'] = sender
msg['To'] = recipient

# Send the email
with smtplib.SMTP('smtp.example.com', 587) as server:
    server.starttls()  # Secure the connection
    server.login('your_email@example.com', 'your_password')
    server.send_message(msg)

print('Email sent!')

• smtplib.SMTP(): Connects to an SMTP server to send the email.
• MIMEText: Creates a text-based email.

Automating File Downloads

You can automate the process of downloading files from the web using Python’s requests module.

Example: Downloading a File

import requests

# URL of the file to download
url = 'https://example.com/file.txt'

# Send a GET request to download the file
response = requests.get(url)

# Save the file locally
with open('file.txt', 'wb') as file:
    file.write(response.content)

print('File downloaded!')

In this example, the file is downloaded from the specified URL and saved locally as file.txt.

Key Concepts Recap

In this lesson, we covered:

• Automating file and directory operations using os and shutil.
• Running system commands and managing environment variables with os.
• Automating web interactions using Selenium, including navigating websites and filling out forms.
• Automating email tasks using smtplib.
• Automating file downloads using requests.

Python’s versatility and extensive library ecosystem make it a powerful tool for automating a wide range of tasks, reducing manual effort, and increasing efficiency.

Exercises

1. Write a Python script that automatically organizes files in a directory based on file extensions (e.g., move all .txt files to a TextFiles folder).
2. Use Selenium to open a website, fill out a search form, and print the results to the console.
3. Automate sending an email with an attachment using `smtplib`.
4. Write a script that downloads a file from the internet and saves it to a specific directory, using the requests module.

FAQ

Q1: What is the difference between using os.system() and the subprocess module for running system commands?

A1:

• os.system(): This is a simpler method to execute system commands, but it only returns the command’s exit status and provides no detailed output control.
• subprocess: This module is more powerful and flexible. It allows you to capture the command’s output (stdout and stderr), interact with processes, and control input/output streams.

Example using subprocess to capture output:

import subprocess

result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)  # Prints the output of the command

Use subprocess for more advanced use cases where you need control over input/output or error handling.

Q2: How can I ensure my automation scripts handle exceptions, like missing files, without crashing?

A2: You can handle exceptions in your script using try-except blocks. This allows your script to continue running even if an error occurs.

Example:

try:
    with open('nonexistent_file.txt', 'r') as file:
        data = file.read()
except FileNotFoundError as e:
    print(f"Error: {e}")

In this case, if the file doesn’t exist, the error is caught, and the script can proceed without crashing.

Q3: How can I automate tasks that require user input, such as filling out interactive command-line prompts?

A3: You can use the subprocess module’s Popen() to interact with command-line prompts by providing input programmatically.

Example:

import subprocess

# Example of passing input to a command-line prompt
process = subprocess.Popen(['python3', 'interactive_script.py'], stdin=subprocess.PIPE, text=True)
process.communicate(input='your_input_here\n')

This method simulates user input, allowing automation of tasks that would normally require user interaction.

Q4: What should I do if my automated tasks involve sensitive information like passwords?

A4: Avoid hardcoding sensitive information (e.g., passwords, API keys) in your scripts. Instead:

1. Use environment variables: Store sensitive data as environment variables and retrieve them in your script using os.getenv().

   import os
   password = os.getenv('MY_PASSWORD')

2. Use a configuration file: Encrypt sensitive information in a configuration file and load it securely.
3. Use secrets management services: For larger projects, use dedicated secrets management tools (e.g., AWS Secrets Manager or HashiCorp Vault).

Q5: How can I automate interactions with websites that have CAPTCHA?

A5: CAPTCHA is designed to block automated scripts. If you encounter CAPTCHA while automating web tasks, consider the following:

1. Manual interaction: You can configure your Selenium script to pause and manually solve the CAPTCHA.
2. Use CAPTCHA-solving services: There are third-party services that can automatically solve CAPTCHAs, but these are often paid and should be used sparingly.
3. Look for alternative data sources: If possible, use an API or another method that does not involve CAPTCHA.

Q6: How do I avoid getting blocked while automating web scraping tasks with Selenium?

A6: To prevent being blocked by websites when using Selenium:

1. Respect robots.txt: Check the website’s robots.txt file to understand which areas can be scraped.
2. Add delays between actions: Use time.sleep() to add delays between requests to mimic human activity.

   import time
   time.sleep(2)  # Wait for 2 seconds before the next action

3. Use a random User-Agent header: Rotate User-Agent strings to avoid detection.
4. Limit requests: Avoid sending too many requests in a short period, as this can trigger rate-limiting or blocking mechanisms.

Q7: How can I automate sending emails with attachments?

A7: You can automate sending emails with attachments using smtplib and email.mime modules.

Example:

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders

# Email details
msg = MIMEMultipart()
msg['From'] = 'your_email@example.com'
msg['To'] = 'recipient_email@example.com'
msg['Subject'] = 'Subject with Attachment'

# Attach a file
filename = 'file.txt'
with open(filename, 'rb') as attachment:
    part = MIMEBase('application', 'octet-stream')
    part.set_payload(attachment.read())
    encoders.encode_base64(part)
    part.add_header('Content-Disposition', f'attachment; filename={filename}')
    msg.attach(part)

# Send the email
with smtplib.SMTP('smtp.example.com', 587) as server:
    server.starttls()
    server.login('your_email@example.com', 'your_password')
    server.sendmail(msg['From'], msg['To'], msg.as_string())

print('Email sent!')

This script attaches a file (file.txt) to the email and sends it via SMTP.

Q8: How can I schedule my automation scripts to run periodically, like once a day?

A8: There are several ways to schedule Python scripts to run periodically:

1. Cron Jobs (Linux/macOS): Use cron to schedule scripts. For example, to run a script daily at 8 AM:

   0 8 * * * /usr/bin/python3 /path/to/script.py

2. Task Scheduler (Windows): Use the Windows Task Scheduler to set up the script to run at regular intervals.
3. Python schedule library: Use the schedule library to schedule tasks from within your Python script.

   import schedule
   import time

   def job():
       print("Task running...")

   schedule.every().day.at("08:00").do(job)

   while True:
       schedule.run_pending()
       time.sleep(1)

Q9: How do I automate a process that requires interacting with GUI applications?

A9: For automating tasks that require interacting with desktop applications, use Python libraries like PyAutoGUI or Pywinauto. These libraries allow you to simulate mouse clicks, keypresses, and interact with on-screen elements.

Example using PyAutoGUI to automate a mouse click:

import pyautogui

# Move the mouse to the x, y position (100, 100) and click
pyautogui.moveTo(100, 100)
pyautogui.click()

This simulates a mouse click at the specified screen coordinates.

Q10: How do I download files from the web if they are behind authentication?

A10: To download files that are behind an authentication wall, use Python’s requests module along with authentication credentials. You can handle Basic Auth or session-based authentication.

Example with Basic Authentication:

import requests

url = 'https://example.com/protected-file'
auth = ('username', 'password')

response = requests.get(url, auth=auth)
with open('downloaded_file.txt', 'wb') as file:
    file.write(response.content)

print('File downloaded!')

For websites that use session-based authentication (like a login form), use requests.Session() to log in and maintain the session across multiple requests.