Testing
Dagster believes that data engineering should feel like software engineering. And anyone who has been involved in software engineering knows the importance of testing.
With proper testing, you can discover issues much earlier in the development process. This helps to minimize risk as you are building out your platform and gives developers more confidence iterating. For those primarily responsible for building pipelines you can spend less time on maintenance and more time on new features. Tests enable easier changes for both team members who have forgotten parts of the codebase and people new to the team and still learning about the edge cases and business logic.
Types of testing
There are many different types of testing. Each focuses on a specific aspect of engineering and requires its own unique approach.
| Type | Description |
|---|---|
| Unit testing | Tests individual components or functions in isolation |
| Integration testing | Ensures that different modules or services work together as expected |
| Data quality testing | Validate data and checks that it meets predefined criteria |
These are the types of tests we will cover in this module but there are many other testing disciplines. Depending on your role within an organization you may be more concerned with different aspects of testing. It is usually not the responsibility of one person to manage the entire testing stack but every role has a hand in testing.