On January 16, 2025, data professionals and enthusiasts gathered at Palais Eschenbach in Vienna and online via Zoom for the latest TDWI e.V. Roundtable, hosted by TCI Consult GmbH. The theme, “Data Quality in Focus: Machine Learning and AI for Data Warehouses & Anomaly Detection,” brought together industry leaders to share insights on leveraging advanced technologies to solve persistent data challenges.

digna was proud to participate in the event, represented by Marcin Chudeusz, who presented insights on “Anomaly Detection in Data Warehouses.” Marcin's presentation showcased digna's innovative approach to leveraging aggregated metrics for anomaly detection, with a live demo that demonstrated how our technology makes data quality management more efficient, scalable, and actionable.

Here are the key takeaways from this engaging session and why it matters for organizations focused on building trust in their data-driven strategies.

Why Anomaly Detection in Data Warehouses Matters

Marcin emphasized the growing importance of data warehouses as the backbone of business analytics. However, ensuring trust in analytics requires a proactive approach to identifying and resolving data quality issues. This is where anomaly detection plays a transformative role by:

• Improving Efficiency: Faster and less resource-intensive analysis of aggregated metrics.

• Integrating Seamlessly: Easily embedded in ETL processes for ongoing monitoring.

  
  

• Identifying Global Patterns: Detecting anomalies at an aggregated level to highlight significant trends.

  
  

• Scalability: Performing effectively even with large datasets.

The Role of Aggregated Metrics in Anomaly Detection

One of the session's highlights was Marcin's explanation of how aggregated metrics provide a robust foundation for anomaly detection. He outlined key metrics used in numerical and categorical columns, such as:

• Count of missing values

• Average and sum of numerical values

• Frequency and uniqueness of categorical values

These metrics streamline the process, reducing false positives and enabling businesses to focus on relevant trends and problems.

Three-Step Process for Effective Anomaly Detection

Marcin shared digna’s three-step process for anomaly detection:

1. Metric Profiling: Data is monitored over time, capturing key statistics like missing values, averages, and unique counts.

   
   

2. AI-Powered Forecasting: Machine learning models predict future metric values using signature-based methods.

   
   

3. Autothreshold Optimization: Thresholds are automatically adjusted using conformal inference, ensuring optimal anomaly detection accuracy.

Real-World Applications of Anomaly Detection

Marcin illustrated how anomaly detection can be applied in data quality assurance and data analytics:

1.Data Quality:

• Verifying incoming data quality.

• Assessing data integrity post-ETL.

• Ensuring the accuracy of generated reports.

2.Data Analytics:

• Detecting revenue aggregation anomalies.

• Monitoring user activity patterns.

• Identifying potential hacker attacks early.

He also provided examples of specific data quality issues that anomaly detection can address, such as missing values, swapped columns, truncated data, and delayed data delivery.

Why This Matters for Organizations

Organizations often face challenges such as resource inefficiency, inconsistent data quality, and delayed issue detection. Marcin’s presentation underscored how proactive anomaly detection addresses these challenges by enabling:

• Early Detection of Data Issues: Avoiding downstream problems in analytics and reporting.

  
  

• Improved Trust in Analytics: Providing reliable data for confident decision-making.

  
  

• Operational Efficiency: Automating processes that traditionally require manual intervention.

Thank You to TDWI and TCI Consult GmbH

We extend our gratitude to TDWI e.V. and TCI Consult GmbH for organizing and hosting this insightful roundtable. Events like these provide invaluable opportunities for data professionals to share knowledge and explore the latest innovations shaping the industry.

Conclusion: A Call to Action for Data Professionals

The TDWI Roundtable highlighted the growing importance of anomaly detection in ensuring data trust and business efficiency. With increasing data volumes and complexities, leveraging innovative approaches like aggregated metrics and AI is no longer optional—it’s essential.

At digna, we are proud to lead this charge, offering solutions that empower data teams to proactively address quality issues and focus on what matters most: driving business value.

Experience the digna Difference

Want to see how our solutions can revolutionize your data quality management? Schedule a demo with digna today and discover the future of anomaly detection and data quality.