With the explosion of data in modern warehouses, lakes, and lakehouses, organizations need a streamlined, intelligent way to detect issues before they cause disruptions. Automating anomaly detection is crucial for maintaining data quality and ensuring the reliability of your data-driven decisions. It stands at the forefront of predictive analytics, transforming how businesses handle vast amounts of data across different architectures.

While traditional tools require exporting data from your platform for analysis, modern data quality tools like digna keep it simple: analyzing the data right where it resides. No costly migrations, no sluggish workflows. Instead, you get automated anomaly detection that’s fast, efficient, and deeply integrated with your existing data infrastructure.

In this guide, we’ll walk you through the process of setting up a project on digna and configuring tables to start detecting anomalies automatically. From database connections to defining alerts, digna makes it easy for data teams to harness the power of AI-driven data quality management.

How to Create Your First Project on digna

Step 1: Create a New Project

The foundation of all your work in digna is creating a Project. This enables you to combine multiple data sources and organize your efforts into manageable pieces. Think of it as your workspace where all the action happens.

• First, locate the “Projects” button. You’ll use this to either create a new project or browse existing ones.

  
  

• Once you click on “Add New Project”, a form will pop up asking for a project name and a brief description. Be clear and concise in your description, as this will help you keep track of the project’s focus.

  
  

• After filling in these details, click 'Create Project' to set up your new data management workspace.

Step 2: Configure Database Connection

With your project in place, the next step is configuring your connection to the database that contains the data you want digna to analyze.

• Choose your database technology (e.g., PostgreSQL, MySQL, Snowflake) and provide connection details.

  
  

• You can connect via Native Connection or ODBC Connection. For Native Connection, you’ll need to configure details such as:

• Host Address

• Host Port

• Database Name

• Schema Name

• Password

For ODBC, simply toggle to the ODBC mode and add the required properties. Once configured, test the connection to ensure it is functioning correctly before proceeding.

Step 3: Set Profiling Mode

Profiling mode is crucial as it determines how digna interacts with your data. digna offers three distinct “Profiling Modes” to manage how data is processed:

1. Standard Mode: digna calculates metrics directly from the source tables without data duplication. This is the fastest method.

   
   

2. Permanent Mode: digna copies the data for the day into a permanent table and then calculates metrics from there.

   
   

3. Session Mode: digna copies data into a session or temporary table and performs the analysis.

Select the mode that best suits your operational preferences and data management policies. Click “Test Connection” to ensure everything is linked correctly. If successful, you’ll receive a notification confirming the setup.

Now let’s configure the tables you want to analyze.

How to Configure Your Data Tables for Anomaly Detection

Step 1: Add Data Tables

Once your project is created and the database connection is live, the next vital step is to define which data tables digna should monitor.

Select the relevant tables from your database to set them up for analysis. digna allows you to choose specific tables, tailor the types of analysis, and set up snapshot queries for detailed inspections. This could include tables that hold critical data like customer transactions, product inventories, or financial records.

Step 2: Weekly Data Overview

After you’ve configured your tables, digna instantly presents a comprehensive “Weekly Data Overview” upon login. This overview highlights potential anomalies across your datasets, offering immediate insights into data quality.

With just a click, you can navigate through different weeks to observe performance trends and understand your data’s health over time.

Step 3: Investigating Alerts

One of digna’s standout features is its ability to detect issues with pinpoint accuracy. Configure notifications to be alerted of anomalies and data quality issues. If any anomalies arise, digna will issue alerts using a traffic light system:

1. Red: Indicates data issues.

2. Amber: Flags suspicious data.

3. Green: Means everything is optimal.

By drilling down into specific alerts, you can identify problem days, inspect flagged tables, and review the affected data columns. digna even provides data visualization charts that visually showcase where and how the data has deviated from expected patterns.

Step 4: digna’s AI-Defined Thresholds

digna’s AI-driven anomaly detection doesn’t just alert you when something is wrong—it also defines acceptable ranges for data metrics. For example, if the system expects missing values to be between 222 and 503, it will flag deviations beyond this range.

But here’s where the magic happens: Autothresholds. digna’s AI continuously learns from your data, self-adjusting thresholds and proactively predicting future anomalies before they occur.

Understanding digna's Anomaly Detection Process

digna's AI-powered anomaly detection leverages advanced algorithms to identify deviations from expected data patterns. Here's a breakdown of how it works:

1. Data Profiling: digna automatically profiles your data, capturing key metrics and establishing baselines.

   
   

2. Anomaly Detection: Our algorithms continuously monitor data for anomalies, comparing it against established baselines and identifying deviations.

   
   

3. Root Cause Analysis: When an anomaly is detected, digna helps you understand the underlying causes by providing detailed information about the affected data points.

   
   

4. Data Visualizations: digna's intuitive dashboards provide clear visualizations of data health, making it easy to identify anomalies and trends.

Why Automating Anomaly Detection with digna is a Game-Changer

In traditional systems, defining technical rules and manually monitoring data quality is time-consuming and stressful. By automating the process, digna frees your data teams from the burden of constant surveillance. Here’s why that matters:

• Speed: Detect anomalies in real-time, allowing you to address issues before they impact operations.

  
  

• Accuracy: With AI-driven insights, you get fewer false positives and more meaningful alerts.

  
  

• Efficiency: Save time by automating data profiling and anomaly detection—no more manual interventions.

  
  

• Flexibility: Whether your data is in the cloud or on-prem, digna integrates seamlessly, keeping your data in place while analyzing key metrics.

Conclusion

By automating anomaly detection with digna, you can significantly improve data quality, reduce the risk of errors, and make more informed decisions. With features like autometrics, autothresholds, and instant alerts, digna helps you maintain the highest data standards with minimal effort. Our user-friendly platform and powerful AI algorithms make it easy to get started and achieve results.

Your data’s story is always unfolding—let digna help you uncover the hidden patterns and anomalies that lie beneath the surface.

Ready to experience the power of automated anomaly detection? Book a live demo with digna today or see digna live in action and discover how our data observability platform can transform your data management practices.