Jupiter.Market.Elastic
Validates and repairs a geometry if possible, ensuring Elasticsearch geo_shape compliance.
The geometry to validate and repair.
If true, validates coordinates are within WGS84 bounds (only use for WGS84 geometries).
Optional logger for warnings.
A valid geometry, or null if the geometry cannot be repaired.
Ensures geometry meets Elasticsearch geo_shape requirements.
Ensures a polygon meets Elasticsearch requirements (orientation, no self-intersections, valid coordinates).
Validates that all coordinates are within valid WGS84 bounds.
Configuration settings for Elasticsearch connection and indexing behavior.
These settings are typically loaded from appsettings.json under the "Elasticsearch" section.
Gets or sets the Elasticsearch Cloud ID for cloud-based deployments.
Format: "deployment-name:base64-encoded-data"
Gets or sets the API key for Elasticsearch authentication.
This should be a base64-encoded API key with appropriate index permissions.
Gets or sets the name of the Elasticsearch index to use.
jupiter-titles
Gets or sets a value indicating whether to delete the existing index before creating a new one.
WARNING: Setting this to true will permanently delete all existing data in the index.
Use with caution, typically only in development or for full re-indexing scenarios.
Gets or sets a value indicating whether to create the index if it doesn't exist.
Gets or sets the number of documents to process in each batch operation.
Typical values: 500-5000. Higher values = faster indexing but more memory usage.
Gets or sets the indexing mode (full, incremental, planning, missing).
Valid values: "full", "incremental", "planning", "missing"
Gets or sets the Elasticsearch query JSON for the elastic-query command.
This should contain a valid Elasticsearch query DSL JSON that will be used
to filter documents when using the 'elastic-query' command.
Configuration settings for PostgreSQL database connection.
These settings are typically loaded from appsettings.json under the "Postgres" section.
Gets or sets the PostgreSQL connection string.
Host=localhost;Database=jupiter;Username=user;Password=pass;SSL Mode=Require;
Gets or sets the maximum number of concurrent database connections allowed.
Default is 5. Adjust based on database server capacity and application load.
Higher values allow more parallel queries but consume more database resources.
Configuration settings for the indexer service behavior.
These settings are typically loaded from appsettings.json under the "Indexer" section.
Gets or sets the file path for storing the last processed title number (offset tracking).
This file enables resumable indexing by tracking progress. Default: "./offset.log"
Gets or sets how frequently to log progress (every N messages processed).
For example, if set to 100, progress will be logged every 100 messages.
Gets or sets the maximum number of concurrent Service Bus message processing operations.
Default is 10. Controls how many titles can be indexed in parallel.
Gets or sets the hour (0-23) when normal queue processing should start.
Default is 20 (8 PM). High-priority queue runs 24/7, normal queue runs during this window.
Gets or sets the hour (0-23) when normal queue processing should end.
Default is 8 (8 AM). Creates a processing window from NormalQueueProcessingStartTime to this hour.
Gets or sets the file path containing title IDs for manual reprocessing.
File should contain one title ID (GUID) per line.
Gets or sets the maximum number of query metrics to retain in memory.
Default is 50. When query metrics exceed this count, the least frequently used metrics are removed.
Higher values track more queries but consume more memory.
Gets or sets how frequently to clean up query metrics (every N messages processed).
Default is 10000. Cleanup runs after processing this many messages to prevent unbounded memory growth.
Represents the status of a message processed during indexing operations.
Gets or sets a value indicating whether the operation was successful.
Gets or sets a descriptive message about the operation result.
Gets or sets the action taken for the message (e.g., Completed, DeadLettered).
Tracks the last processed Service Bus message for resumable operations.
Gets or sets the last modified date processed.
Gets or sets the last created date processed.
Gets or sets the last processed title ID.
Gets or sets a value indicating whether the initial null modified date phase has been processed.
Configuration settings for Azure Service Bus integration.
Gets or sets the Service Bus connection string.
Gets or sets the name of the Service Bus topic for title updates.
Gets or sets the name of the normal priority subscription.
Gets or sets the name of the high-priority subscription.
Gets or sets the file path for tracking the last processed Service Bus message.
Gets or sets the name of the Service Bus topic for deal updates.
Gets or sets the maximum duration for which the message lock will be renewed automatically.
Default is 10 minutes. This prevents MessageLockLost exceptions for long-running message processing.
Represents the base reference information for a property title retrieved from PostgreSQL.
This class is used as an intermediate data transfer object when retrieving title data
from the database before mapping to the full Title object for Elasticsearch indexing.
Represents a complete property title document for Elasticsearch indexing in the .
This is the root document type indexed in Elasticsearch. It contains all information
about a property title including:
- Basic title information (number, area, tenure) from
- Address details from
- Historical sales data from
- Ownership information from
- Buildings from and UPRNs from (nested hierarchy)
- Planning applications from
- Land constraints from
- Lease information from
- Geospatial data (polygon geometry and centroid) from
The class uses attributes to define Elasticsearch field mappings.
See also: Data Model
Gets or sets the unique identifier for the title.
This GUID serves as the Elasticsearch document ID and primary key.
Gets or sets the official title number (e.g., "DN123456").
This is the Land Registry title number and is indexed as full-text searchable.
Gets the total building footprint in square feet by summing all building footprints.
This is a calculated property that aggregates footprint areas from all buildings on the title.
Returns 0 if there are no buildings.
Gets the site coverage percentage (building footprint / total site area * 100).
This calculated property indicates what percentage of the site is covered by buildings.
Returns null if building footprint or area is not available or area is zero.
Postcode as structured feature (aligned with ).
Generic planning use class description from NewPlanningUseCode (e.g. "General industry").
Specific AddressBase/VOA description from Classification (e.g. "Recycling Site").
Represents a minimal projection of a title for queueing and indexing operations.
Gets or sets the unique identifier for the title.
Gets or sets the last modified date of the title record.
Gets or sets the created date of the title record.
Gets or sets the processing priority for the title (e.g., High, Medium).
Represents a message payload for Service Bus title update events.
Gets or sets the unique identifier for the title.
Gets or sets the processing priority for the message (default: Medium).
Represents a projection containing only the title number for lightweight queries.
Gets or sets the official title number (e.g., "DN123456").
Provides data access services for retrieving and processing title information from PostgreSQL.
Handles connection management, query execution, retry logic, and performance tracking.
This service uses Dapper for data access, Polly for retry policies, and NetTopologySuite for geospatial data.
All queries support cancellation and optional retry logic for transient failures.
See also: Data Retrieval Guide
Gets the concurrent dictionary tracking performance metrics for all executed queries.
Metrics are tracked by query key and include min, max, average execution times and run counts.
Cleans up query metrics that haven't been used recently to prevent unbounded memory growth.
Keeps metrics for queries that have been run recently or have significant data.
Maximum number of query metrics to retain. Defaults to 50.
Tracks performance metrics for database queries including execution times and run counts.
Gets or sets the minimum query execution time in milliseconds.
Gets or sets the maximum query execution time in milliseconds.
Gets or sets the total accumulated execution time in milliseconds.
Gets or sets the total number of times this query has been executed.
Gets the average query execution time in milliseconds.
Updates the metrics with a new query execution duration.
The execution duration in milliseconds.
Initializes the DataService by setting up the PostgreSQL connection pool and verifying PostGIS connectivity.
A task representing the asynchronous initialization operation.
Thrown when initialization fails after retries.
This method:
- Creates an NpgsqlDataSource with NetTopologySuite support for geospatial data
- Configures type mappings (numeric to double)
- Verifies PostGIS is available by executing a version query
- Retries up to 3 times with exponential backoff (2s, 4s, 6s) for Azure PostgreSQL readiness
Opens and returns a new PostgreSQL connection from the connection pool.
An open NpgsqlConnection ready for use.
Thrown when connection cannot be established.
Connections are managed by the NpgsqlDataSource pool and should be disposed after use.
Executes a SQL query and returns the results as a collection of typed objects.
The type to map query results to.
The SQL query to execute.
Anonymous object containing query parameters.
If true, applies Polly retry policy for transient failures.
Cancellation token for the operation.
Optional key for performance tracking.
A collection of objects of type T.
Thrown when query fails after retries (if enabled).
Thrown when query times out after retries (if enabled).
When useRetry is true, the query will be retried up to 3 times with exponential backoff (2s, 4s, 8s)
for transient PostgreSQL exceptions and timeouts. Command timeout is set to 90 seconds.
var sales = await RunQueryAsync<Jupiter.Market.Elastic.Titles.Models.Sale>(
"SELECT * FROM market.\"Sale\" WHERE \"TitleId\" = @TitleId",
new { TitleId = titleId },
useRetry: true,
cancellationToken
);
Executes a pre-defined query from QueryConstants and tracks its performance metrics.
The type to map query results to.
The key identifying the query in QueryConstants.Queries dictionary.
Anonymous object containing query parameters.
If true, applies Polly retry policy for transient failures.
Cancellation token for the operation.
A collection of objects of type T.
Thrown when queryKey is not found in .
Thrown when query fails after retries (if enabled).
This method automatically tracks query performance metrics (min, max, average execution time, run count)
in the QueryMetrics dictionary. Metrics are updated in a thread-safe manner.
var sales = await RunQueryByKeyAsync<Jupiter.Market.Elastic.Titles.Models.Sale>(
"sales",
new { TitleId = titleId },
useRetry: true,
cancellationToken
);
Retrieves all UPRNs (Unique Property Reference Numbers) for a title with comprehensive enrichment data.
The unique identifier of the title.
Cancellation token for the operation.
A dictionary mapping building IDs to lists of enriched UPRN objects.
This method executes a complex CTE query that:
- Identifies all UPRNs associated with the title and their parent buildings from
- Aggregates rateable values from multiple assessments from
- Selects the most reliable size data based on source priority (VOA > EPC > RM > ZP > AB)
- Selects the most reliable room counts based on source priority
- Retrieves the latest EPC (Energy Performance Certificate) ratings from
- Aggregates VOA classifications as JSON from
- Aggregates planning use classes as JSON
- Concatenates floor levels in sorted order
The returned dictionary groups UPRNs by their parent building ID. UPRNs without buildings
are assigned to a zero GUID building ID.
See also: Data Retrieval Guide
var uprnsByBuilding = await GetUprnsByTitleIdsAsync(titleId, cancellationToken);
foreach (var (buildingId, uprns) in uprnsByBuilding)
{
Console.WriteLine($"Building {buildingId} has {uprns.Count} UPRNs");
}
Retrieves the base title reference information including area measurements and address details.
The unique identifier of the title.
Cancellation token for the operation.
A object containing base title information, or null if not found.
This method retrieves core title information from including:
- Title number and areas (square meters, square feet, acres)
- Tenure type (Freehold, Leasehold)
- Property category and types
- Complete address components from (building number, street, town, county, postcode)
The method handles flexible PropertyType mapping, supporting List<string>, string[],
or single string values from the database.
See also: Data Retrieval Guide
var titleRef = await GetTitleReferenceByIdAsync(titleId, cancellationToken);
if (titleRef != null)
{
Console.WriteLine($"Title: {titleRef.Number}, Area: {titleRef.AreaInSquareFeet} sq ft");
}
Provides services for managing Elasticsearch index operations including document indexing,
index creation/deletion, data validation, and efficient hash-based change detection for upserts.
This service uses the NEST client library to interact with Elasticsearch Cloud.
It handles index lifecycle management, bulk operations, and maintains synchronization
between PostgreSQL and the .
Efficient upserts are performed using a hash-based change detection strategy:
a deterministic hash is computed for each document and stored in Elasticsearch. Only documents
with missing or changed hashes are re-indexed, avoiding unnecessary updates and per-document GETs.
See also: Architecture Overview
Initializes the Elasticsearch client with cloud connection settings and default mappings.
A task representing the asynchronous initialization operation.
This method configures the NEST client with:
- Cloud connection pool using CloudId and API key authentication
- Default index name from settings
- Debug mode for detailed logging
- Custom field name inference (camelCase)
- Default mapping for type
See also: Configuration Guide
Upserts (inserts or updates) a single title document in the using hash-based change detection.
Also sends a Deal update message to the Service Bus with a randomized priority ("High" or "Medium") after a successful upsert.
The object to index.
A if the document was indexed, or null if no changes were detected.
This method computes a deterministic hash for the document and delegates to .
It avoids per-document GETs and deep JSON comparisons by using a hash field (DocHash) stored in Elasticsearch.
If the hash matches the existing document, no re-indexing is performed.
After a successful upsert, a message is sent to the Deal Service Bus topic to notify the Deal service of the update.
The message includes the Deal ID and a randomized priority ("High" or "Medium").
See also: Data Model
var response = await UpsertIndex(title);
if (response != null)
{
_logger.LogInformation($"Indexed title {title.Id}");
}
else
{
_logger.LogInformation($"No changes detected for title {title.Id}, skipping re-index.");
}
// A Deal update message is also sent to the Service Bus with randomized priority.
Bulk upserts title documents using hash-based change detection.
List of documents to upsert.
A from Elasticsearch, or null if no documents required indexing.
This method computes a deterministic hash (DocHash) for each document using a canonical JSON representation.
It then fetches existing hashes from Elasticsearch in a single ids query (source filtering only for ID and DocHash).
Only documents whose hash is missing or different are included in the bulk index operation.
This avoids unnecessary re-indexing and eliminates per-document GETs or deep JSON comparisons.
The hash is stored in the DocHash field of each document in Elasticsearch.
Deletes a title document from the .
The unique identifier of the title to delete.
A task representing the asynchronous delete operation.
This method silently handles exceptions to prevent failures when deleting non-existent documents.
Used to clean up stale documents when titles are removed from .
Queries Elasticsearch using the query from configuration and sends matching document IDs to Service Bus.
Cancellation token for the operation.
A task representing the asynchronous operation.
This method:
- Parses the QueryJson from ElasticsearchSettings configuration
- Executes a scroll query against the Elasticsearch index
- Streams document IDs in batches to Service Bus via ServiceBusMessageService
- Logs progress periodically
The query can potentially match millions of documents and uses Elasticsearch scroll API
for efficient streaming without loading all results into memory.
Defines actions for handling Azure Service Bus messages in the indexing workflow.
Completes the specified Service Bus message, removing it from the queue.
The message to complete.
A cancellation token.
Abandons the specified message, making it available for reprocessing.
The message to abandon.
Optional properties to modify on the message.
A cancellation token.
Moves the specified message to the dead-letter queue with a reason and optional description.
The message to dead-letter.
The reason for dead-lettering.
An optional error description.
A cancellation token.
Defers the specified message for later processing.
The message to defer.
Optional properties to modify on the message.
A cancellation token.
Orchestrates the indexing process for property titles, managing Service Bus message processing,
data retrieval, and Elasticsearch indexing operations.
This service coordinates between (PostgreSQL), (Elasticsearch),
and (Azure Service Bus) to process title updates in real-time or batch mode.
Supports high-priority and regular message processing with configurable time windows.
See also: Architecture Overview
Indexes a single title by retrieving all related data from PostgreSQL and upserting it into the .
The unique identifier of the title to index.
Cancellation token for the operation.
A tuple containing:
- bool: True if indexing succeeded, false otherwise
- string: Error message if indexing failed, empty string if successful
This method performs the following steps:
- Retrieves the base title reference from
- Fetches all related data in parallel (sales from , owners from , planning from , buildings from , UPRNs from , geometry from , leases from , constraints from )
- Processes geometry data (coordinate transformation, centroid calculation)
- Builds the building-UPRN hierarchy
- Constructs the complete object
- Upserts the document into Elasticsearch
If the title is not found in PostgreSQL, any existing Elasticsearch document is deleted
and the method returns success (not an error condition).
All data retrieval operations execute in parallel for optimal performance.
See also: Title Construction Process
var (success, errorMessage) = await IndexTitleByIdAsync(titleId, cancellationToken);
if (success)
{
_logger.LogInformation("Successfully indexed title {TitleId}", titleId);
}
else
{
_logger.LogError("Failed to index title {TitleId}: {Error}", titleId, errorMessage);
}
Provides deterministic, canonical hash computation for documents using JSON serialization.
The hash is computed using SHA-256 over a canonical JSON representation of the document:
- Property names are camelCase
- Null values are ignored
- Enums are serialized as camelCase strings
- Property order is stable
- No indentation or extra whitespace
This is used for efficient change detection in Elasticsearch upserts.
The canonical JSON serializer options for hash computation.
Computes a deterministic SHA-256 hash for the given document using canonical JSON serialization.
The type of the document.
The document to hash.
A lowercase hex string representing the SHA-256 hash of the canonical JSON.