Herdwatch Enables Customer-Facing Analytics Using Apache Iceberg and PhoenixAI Cloud

About Herdwatch

At Herdwatch, our mission is to empower farmers across Europe with a platform that simplifies farm compliance, tracks animal lifecycle data, and improves operational efficiency. By centralizing information on livestock health, medication schedules, and farm productivity, we help farmers make data-driven decisions.

This article was originally written by Alfred Johnson, Head of Data at Herdwatch.

As our operations expanded across Ireland, the UK, and beyond, we needed a scalable, unified data platform that could deliver real-time insights to our customers and handle growing analytics workloads.

The challenge

Siloed data and slow dashboards

In the past, our analytics workloads relied on region-specific MySQL RDS databases. This worked initially, but it soon created major issues:

• Siloed analytics: Each region had its own database, so users had to consult separate dashboards for insights. Unified reporting was impossible.
• Performance bottlenecks: Dashboards querying RDS replicas were slow, prone to timeouts, and lacked scalability, which made them unsuitable for customer-facing dashboards.

We knew it was time for a significant overhaul of our architecture.

Apache Iceberg to remove the silos

We decided to rebuild our data foundation with Apache Iceberg as the backbone. Iceberg gave us the ability to unify regional datasets and build a scalable, efficient analytics framework on top of them.

Athena: limited success

We first tried AWS Athena as the analytics engine for querying Iceberg data. Its serverless model let us get started quickly, but we ran into several issues:

• High latency: Dashboard queries often took 2–5 minutes to load, far too slow for interactive, customer-facing analytics.
• Limited configurability: Athena's serverless design left us little room for query optimization.
• Cost inefficiency: With frequent, complex dashboard queries, Athena's pay-per-scan model became increasingly expensive.

We needed a solution that met our performance and cost requirements.

The solution

We then found StarRocks, a query engine for customer-facing analytics on open table formats such as Apache Iceberg and Delta Lake. After thorough testing, we adopted an architecture that combines Iceberg and StarRocks:

• Centralized data lake: All regional pipelines store data in Iceberg tables on AWS S3, giving us a single source of truth.
• ETL pipelines: AWS Glue and dbt transform raw data into bronze, silver, and gold layers, handling transformations and aggregations.
• StarRocks as the query layer: StarRocks accelerates queries on the gold layer and powers both internal BI tools and customer-facing applications.
• Materialized views: For complex queries, materialized views pre-aggregate data.
• Customer-facing dashboards: Dashboards query StarRocks directly through its MySQL-compatible interface, with sub-second response times.

We run StarRocks on PhoenixAI Cloud, the cloud-managed version of StarRocks, which adds features such as multi-warehouse support and advanced security controls.

The results

Unified analytics

• We consolidated all regional datasets into a single source of truth, which enables unified reporting and simplifies governance.
• We removed the complexity of region-specific dashboards.
• The same platform handles customer-facing dashboards and internal BI.

Improved performance

• Query latency dropped from 2–5 minutes (Athena) to 700 milliseconds–1.5 seconds (StarRocks).
• Dashboards with multiple queries per page load in under a second.

Cost savings

• We moved from Athena's pay-per-scan model to StarRocks' caching, which minimizes S3 scan costs.
• Materialized views reduce compute use by cutting down on-the-fly joins.

Scalability

• The platform supports analytics for millions of livestock records across thousands of farms, with room to grow.

Operational efficiency

• Iceberg and StarRocks' MySQL compatibility unified our stack and reduced maintenance overhead.
• Centralized data governance and lineage made developer onboarding simpler.

What's next

• IoT integration: Bringing in real-time telemetry from wearable livestock devices for instant insights on health, location, and productivity.
• Querying Iceberg directly: Exploring StarRocks' ability to query Iceberg tables directly, to simplify the architecture further and reduce latency.
• Advanced use cases: Scaling to support more complex analytics for enterprise customers.