13 Skill 7 - Managing data and files, and testing the software solution

SAT701: Organises and manipulates appropriate data structures efficiently to manage data and files.
SAT702: Proposes and implements procedures to manage the security of data and files.
SAT703: Documents the use of testing techniques and test data.

13.1 SAT701: Organises and manipulates appropriate data structures efficiently to manage data and files

🚧 Database
🚧 Explain how files are organised in Django
🚧 Static File Handling: static files such as images, CSS, and JavaScript files
🚧 Media Files Handling: Media files are user-uploaded files such as images, videos, and documents.

13.1.1 Present and explain how files are organised in your Django project

13.2 SAT702: Proposes and implements procedures to manage the security of data and files.

13.2.1 Show Django File Structure

.
├── crm
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-312.pyc
│   │   ├── admin.cpython-312.pyc
│   │   ├── apps.cpython-312.pyc
│   │   ├── models.cpython-312.pyc
│   │   ├── urls.cpython-312.pyc
│   │   └── views.cpython-312.pyc
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __pycache__
│   │       └── __init__.cpython-312.pyc
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── db.sqlite3
├── manage.py
├── mvp24
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-312.pyc
│   │   ├── settings.cpython-312.pyc
│   │   ├── urls.cpython-312.pyc
│   │   └── wsgi.cpython-312.pyc
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── templates
    ├── base.html
    ├── crm
    │   └── home.html
    ├── navbar.html
    └── registration
        └── login.html

Django Project Structure Research

Project Structure
- What is the purpose of the manage.py file in a Django project?
- Identify the location where the database file is stored in this project.
- What is the significance of the __init__.py files found in the directories?
Applications
- What are the names of the applications in this Django project?
- Where would you place new models for the CRM application?
- What is the purpose of the admin.py file in the CRM application?
Static and Template Files
- Where are the HTML templates for the CRM application located?
- What is the purpose of the base.html template?
Migrations
- What is the role of the migrations directory in the CRM application?
- Explain the significance of the __init__.py file within the migrations directory.
Django Configuration
- What files in the mvp24 directory are responsible for configuring the Django project?
- Where are the URL configurations defined for this project?
Compiled Python Files
- What is the purpose of the __pycache__ directories found throughout the project?
- Why are there .pyc files in the __pycache__ directories?
User Authentication
- Where is the login template for user authentication located in this project?
Views and URLs
- In which files would you define views and URLs for the CRM application?
- How would you register a new URL route for the CRM application?
Testing
- Where would you write tests for the CRM application?
Deployment
- What is the purpose of the wsgi.py and asgi.py files in the mvp24 directory?
Virtual Environment
- What is a virtual environment, and why is it used in a Django project?
- Explain the significance of the .venv directory, even though it is not shown in the tree structure.

13.2.2 Version Control wtih PyCharm

Video Tutorial: Git Tutorial for Beginners

Instructions for Students:

Watch the video tutorial linked above.
Follow along with the steps demonstrated in the video.
Answer the questions and complete the tasks below as you watch the video.

🤔Check Your Understanding

Explained what Git is and its usefulness.
Listed key Git commands and their purposes.
Initialized a Git repository.
Checked the status of the repository.
Added files to the staging area.
Committed changes with a message.

13.3 702 Git Practical

13.3.1 How Git Works

More info:

13.3.2 Git with PyCharm

Review Key ideas:

Code storage locations:

Working directory
Staging area
Local repository
Remote repository

Basic Git workflow:

git clone: Clone a remote repository
git add: Stage changes
git commit: Commit changes
git push: Push commits to remote
git pull: Fetch and merge remote changes
git checkout: Switch branches
git branch: Create branches

13.3.3 Let’s try!

## Initialize a New Repository
# This command creates a new Git repository in the current directory.
git init

## Clone a Repository
# This command creates a copy of an existing repository from a remote server.
# git clone <repository_url>
# Try this:
git clone https://gitlab.sod.thaac.net/jchen/gitfun


## Check the Status of the Repository
# This command shows the current status of the repository, including any changes that have been staged, committed, or modified.
git status

## Add Files to the Staging Area
# This command adds the specified file(s) to the staging area, preparing them to be committed.
# git add <filename>
# For example:
git add Jeremy.md

## Commit Changes
# This command commits the staged changes to the repository with a message describing the changes.
# git commit -m "Jeremy commit"

## View Commit History
# This command displays the commit history, showing a list of all commits made to the repository.
git log

git log --oneline  # shorter information

## Push Changes to a Remote Repository
# This command pushes the committed changes from your local repository to the remote repository.
# git push origin <branchname>
# For example:
git push origin main

## Pull Changes from a remote Repository
## Pull the latest changes from the remote repository to keep your local repository up-to-date:
git pull origin main

13.4 Git tags

This shows you the concept of git tags.

You can tag specific points in a repository’s history as being important.

After you completed version control of your home page, let’s see the history.

git log --oneline
ec71aeb (HEAD -> main, origin/main) MVP - Home Page

Tag the current code.

git tag v0.1.0

Show all tags.

git tab

Push the tag to the remote repository

git push origin v0.1.0
# if you have manay tags to push
git push origin --tags

More information: https://git-scm.com/book/en/v2/Git-Basics-Tagging

✅First version control v.0.1.0

Now, tag your project (home page) v0.1.0

13.5 Django Security Topics

13.5.1 1. Cross-Site Request Forgery (CSRF)

CSRF Tokens: How Django uses CSRF tokens to protect against CSRF attacks.
(csrf_exempt?): When and why you might use @csrf_exempt and the risks involved.

13.5.2 2. Cross-Site Scripting (XSS)

Escaping User Input: How Django automatically escapes user input in templates to prevent XSS.
Safe Markup: Using mark_safe and understanding when it’s appropriate to use it.

13.5.3 3. SQL Injection

Query Parameterization: How Django ORM automatically protects against SQL injection.
Raw SQL Queries: How to safely execute raw SQL queries in Django.

13.5.4 4. Clickjacking

X-Frame-Options: How to use the X-Frame-Options header to protect against clickjacking.
(xframe_options_exempt?): When to use and the risks involved.

13.5.5 5. Authentication and Authorization

Django’s Authentication System: How to use Django’s built-in authentication system.
Permissions and Groups: How to manage user permissions and groups.
Custom User Models: Creating and managing custom user models.

13.5.6 6. Password Management

Password Hashing: How Django hashes passwords and the algorithms it uses.
Password Validation: Using and customizing Django’s password validation rules.

13.5.8 8. HTTPS

SSL/TLS Configuration: How to set up HTTPS for your Django application.
SECURE_SSL_REDIRECT: Redirecting all HTTP traffic to HTTPS.
SECURE_HSTS_SECONDS: Enforcing HTTP Strict Transport Security (HSTS).

13.5.9 9. Security Middleware

SecurityMiddleware: Configuring Django’s SecurityMiddleware to enforce various security settings.
X-Content-Type-Options: Preventing MIME type sniffing.

13.5.10 10. Sensitive Data Exposure

Environment Variables: Storing sensitive information in environment variables.
django-environ: Using django-environ to manage environment variables.

13.5.11 11. Rate Limiting

Throttling: Implementing rate limiting with Django Rest Framework.
Third-Party Tools: Using tools like django-ratelimit to prevent abuse.

13.5.12 12. Content Security Policy (CSP)

CSP Headers: Implementing Content Security Policy to prevent XSS and data injection attacks.
django-csp: Using the django-csp package to manage CSP headers.

13.5.13 13. File Upload Security

File Validation: Validating uploaded files to ensure they are safe.
Storage Backends: Using secure storage backends for file uploads.

13.5.14 14. Third-Party Packages

Dependency Management: Keeping third-party packages up-to-date to avoid known vulnerabilities.
pip-audit: Using tools like pip-audit to check for vulnerabilities in your dependencies.

13.5.15 15. Logging and Monitoring

Error Logging: Setting up logging to monitor for security issues.
Django Admin Logs: Reviewing Django admin logs for suspicious activity.

13.5.16 16. Secure Deployment

Configuration Management: Using tools like Ansible or Chef to manage secure deployment configurations.
Docker Security: Best practices for securing Django applications in Docker containers.

13.5.17 17. User Input Validation

Form Validation: Ensuring all user input is properly validated before processing.
Model Validation: Using Django model validators to enforce data integrity.

13.5.18 18. Data Encryption

Field Encryption: Encrypting sensitive data fields in the database.
django-fernet-fields: Using packages like django-fernet-fields to add encryption to model fields.

13.5.19 19. API Security

Token Authentication: Using token-based authentication for securing APIs.
OAuth2: Implementing OAuth2 for more secure API authentication.

13.5.20 20. Session Security

Session Timeout: Implementing session timeout policies.
Session Hijacking Protection: Using Django’s built-in session security features.

13.6 SAT703: Documents the Use of Testing Techniques and Test Data

13.6.1 Testing Types

Testing different aspects of a Django application, like Unit, Integration, System, and Usability testing, helps make sure everything works well and is easy to use. Unit Testing checks small parts to catch errors early. Integration Testing makes sure these parts work well together. System Testing checks the whole application to see if it meets the goals set for it. Finally, Usability Testing ensures that the app is easy and pleasant for people to use. These tests together help create reliable and user-friendly software. (Usability Testing will be discussed in SAT Skill 8.)

13.6.2 How to test a function

# test_encryption.py
def encrypt(clear_text, shift):
    #Encrypt the clear text
    code = ""
    for ch in clear_text:
        code += chr((ord(ch)-ord("A")+shift)%26 + ord("A"))
    return code


def test_encrypt():
    # Test case: encrypt "HELLO" with a shift of 13
    assert encrypt("HELLO", 13) == "URYYB"

    # Additional test cases
    # Test case: encrypt "WORLD" with a shift of 5
    assert encrypt("WORLD", 5) == "BTWQI"

    # Test case: encrypt "ABC" with a shift of 26 (should remain the same)
    assert encrypt("ABC", 26) == "ABC"

    # Test case: encrypt "XYZ" with a shift of 3 (wraps around to "ABC")
    assert encrypt("XYZ", 3) == "ABC"

Testing Code

Get pytest by pip install pytest
Run tests: pytest test_encryption.py

$ pytest .\test_encryption.py
============================= test session starts =============================
platform win32 -- Python 3.8.17, pytest-8.3.2, pluggy-1.5.0
rootdir: D:\git\2024\jbooks\acsddev\Algorithms
plugins: anyio-3.7.1
collected 1 item

test_encryption.py .                                                     [100%]

============================== 1 passed in 0.04s ==============================

Explanation

The test function test_encrypt directly uses assert statements to verify the expected output for different inputs.

13.6.3 Testing Record Model by Hand

Test Case ID	Test Description	Steps	Expected Result
TC-01	Verify Home Page	1. Open the browser and navigate to the home page URL (e.g., `/`).	The home page should display with a developer name “Jeremy Chen”.
TC-02	View Record List	1. Navigate to the record list page URL (e.g., `/records/`).	A list of records should display. Each record should show `first_name`, `last_name`, `email`, `phone`, `address`, `city`, `state`, `zipcode`, and `created_at`.
TC-03	View Record Detail	1. Click on a record’s ID or name in the record list page.	The detail page for that record should display, showing all fields of the selected record.
TC-04	Create a New Record	1. Navigate to the create record page (e.g., `/records/create/`). 2. Fill in the form with valid data. 3. Submit the form.	The new record should be created and appear in the record list.
TC-05	Update an Existing Record	1. Navigate to the record detail page for an existing record. 2. Click the “Update” button. 3. Modify the data in the form. 4. Submit the form.	The record should be updated with the new data and appear in the list.
TC-06	Delete a Record	1. Navigate to the record detail page for an existing record. 2. Click the “Delete” button. 3. Confirm the deletion.	The record should be removed from the record list.
TC-07	Verify Access Control	1. Attempt to create, update, or delete a record without logging in.	The system should redirect to the login page, as these actions require authentication.

13.6.4 Testing Django with Pytest

This is fairly complicated so you can just do the tests by hand as above.

Install pytest: pip install pytest-django
Create pytest.ini in the project root directory
Create your tests in tests.py
Run tests: pytest

pytest.ini

[pytest]
DJANGO_SETTINGS_MODULE = mvp24.settings
python_files = tests.py test_*.py *_tests.py

View tests.py here:

https://github.com/jeremy886/mvp24/blob/main/crm/tests.py

pytest
==================== test session starts ====================
platform darwin -- Python 3.12.0, pytest-8.3.2, pluggy-1.5.0
django: version: 5.0.7, settings: mvp24.settings (from ini)
rootdir: /Git/mvp24
configfile: pytest.ini
plugins: django-4.8.0
collected 8 items                                           

crm/tests.py ........                                 [100%]

===================== 8 passed in 0.80s =====================