DUFT Server: The Core Execution Engine

DUFT Server: The Core Execution Engine#

DUFT Server acts as the brain of the system, processing queries, executing ETL tasks, and enforcing authentication. It is built on Django/Python, leveraging Django’s built-in security and API framework. Through a modular approach, DUFT Server remains lightweight and extensible, relying on DUFT Config to determine behaviour instead of hardcoded rules.

Engineering Decisions: DUFT Server#

  • Python and Django as the foundation, leveraging Django’s industry-standard framework for web applications while aligning with Python’s dominance in data engineering and analytics. This ensures that DUFT Server integrates seamlessly with data processing tasks while benefiting from a robust, well-supported ecosystem.

  • Utilise Django’s built-in features such as authentication, ORM, and REST framework to avoid unnecessary custom implementations. This reduces development overhead, enhances security, and ensures compatibility with industry best practices.

  • Modular service-oriented architecture to allow new services to be easily added without affecting the core system. This enables DUFT Server to scale organically, supporting new functionality such as additional data sources, APIs, or workflow automation without major rewrites.

  • Decoupled configuration via DUFT Config, ensuring that the server remains lightweight and adaptable. Instead of hardcoding logic, dashboards, queries, and workflows are externally defined, allowing behaviour to be updated dynamically without modifying the backend.

  • Asynchronous execution support for long-running tasks such as ETL and data uploads. By handling these tasks outside the request-response cycle, DUFT Server remains responsive while allowing for large-scale data transformations and reporting.

Key Services Provided by DUFT Server include:

Query Service

One of the critical components of DUFT Server is the Query Service, which is responsible for executing SQL-based queries that power the dashboards. When a user requests data through DUFT UI, the Query Service fetches the relevant SQL definition from DUFT Config, executes it against the database, and converts the results into a structured JSON format that the frontend can render. To ensure efficiency, the service employs caching and optimised execution strategies, reducing redundant queries and improving response times.

Data Task Engine (DTE)

In addition to handling queries, DUFT Server incorporates the Data Task Engine (DTE), which executes Python scripts and Jupyter notebooks for ETL processes. These Data Tasks facilitate a wide range of data operations, including extracting and transforming information from external sources, performing aggregations and calculations for dashboards, and updating DUFT Config dynamically to modify system behaviour in real-time. This ability to execute scheduled and on-demand data transformations enables DUFT to function as a robust analytics engine.

Config & App Service

To facilitate interaction between DUFT UI and DUFT Server, the Config & App Service provides API endpoints that serve dashboard definitions, navigation structures, and queries stored in DUFT Config. It also manages user session data and application-specific configurations, ensuring seamless integration between the backend and frontend. DUFT introduces the concept of apps, which group related dashboards, queries, and ETL tasks into self-contained modules, such as Monthly Reporting or Clinical Monitoring. This modular approach improves organisation and maintainability while enabling specific teams or use cases to have dedicated sections within the system.

Upload Service

For systems requiring file handling, DUFT Server includes an Upload Service, which supports file uploads when deployed as a cloud service. This service allows Data Tasks to push files to other DUFT instances, such as for aggregating monthly summaries across multiple deployments. Uploaded files are stored on the server and can be processed further by other ETL jobs or automated workflows, ensuring smooth data transfer and integration across different systems.

Security Service

Security within DUFT Server is managed by the Security Service, which oversees authentication and role-based access control using Django’s built-in security framework. While authentication is an optional feature, DUFT can operate in an open-access mode when security restrictions are not required. Future enhancements will allow DUFT Config to define custom roles, enabling fine-grained access control for specific apps and dashboards. This planned feature will integrate seamlessly with Django’s authentication system, allowing administrators to set permissions without modifying the core server logic.

Contents