The primary objective of this sample requirement is to exemplify the implementation of an efficient and streamlined integration solution using the Multi-Channel Framework (MCF). As an open-source integration framework, MCF offers a comprehensive set of tools and features that facilitate seamless integration of disparate systems and data sources.
In this context, the requirement emphasizes the utilization of MCF’s unique capabilities through a “channel configuration JSON” file. This file serves as a central hub for defining integration channels, their specific properties, and the desired behavior for each channel. By leveraging this declarative approach, developers can achieve the desired integration outcomes without the need for manual creation of Apache Camel routes, complex data transformations, or intricate orchestration logic through traditional coding practices.
Instead, the MCF engine, which is seamlessly integrated into the project as a dependency, automatically scans and interprets the provided channel configuration JSON. It intelligently interprets the defined configurations and generates the necessary implementation foundations based on their specifications. This automated process eliminates the need for explicit code-based route creation, transformation coding, or intricate orchestration scripting, resulting in a more streamlined and developer-friendly integration development experience.
By leveraging MCF’s channel configuration approach, organizations benefit from accelerated delivery and deployment of integration solutions. The speed to delivery is enhanced by the elimination of manual coding efforts traditionally associated with integration development. Developers can focus on defining the integration requirements in the channel configuration JSON, which serves as a concise and easily understandable representation of the desired integration behaviors. This approach enables faster iterations, as developers can quickly prototype, test, and refine integration configurations without getting caught up in intricate coding details.
Additionally, the speed to deployment is improved through MCF’s automated implementation generation. Once the channel configuration JSON is provided, the MCF engine takes care of the implementation details, automatically generating the underlying integration components such as Apache Camel routes, transformations, and orchestration logic. This automation reduces the time and effort required for deploying integration solutions, as developers can rely on the MCF engine to handle the technical intricacies of translating channel configurations into functioning integration components.
The combined effect of faster delivery and deployment translates into tangible benefits for organizations. It allows them to respond swiftly to changing business needs, evolving market dynamics, and emerging opportunities. With reduced development and deployment cycles, organizations can expedite time-to-market for their integration solutions, gaining a competitive advantage in a fast-paced business landscape.
Development Requirement: Send Event Data to the ISX Backend
Background:
Our organization requires a data integration solution to receive events from multiple sources into our transportation infrastructure central system, ISX, for further processing and analysis. These events need to be received and pushed to our internal Kafka and ActiveMQ data fabric. Additionally, a copy of these events must be sent simultaneously to analysis platforms like AWS OpenSearch or Elasticsearch/Kibana. To accomplish this, we have chosen to implement the solution using the Multi-Channel Framework (MCF), an open-source integration framework. MCF provides the necessary capabilities to seamlessly handle the event ingestion, distribution to internal data fabric, and forwarding to the analysis platforms, enabling efficient data integration and analysis
Functional Requirements:
- Create a RESTful API endpoint to receive JSON formatted events or consume XML events from ActiveMQ.
- Secure the RESTful API with OAuth.
- Transform incoming events from XML to JSON if from ActiveMQ.
- Send event to Kafka and ActiveMQ ISX data fabric backed and AWS OpenSearch simultaneously .
- Send response back it RESTful endpoint.
MCF REST Configuration Sample:
{
"channel_id": "mcf-channel-isx-data-rest",
"channel_src_id": "",
"channel_type": "rest",/*Rest-From*/
"channel_metric_enabled": "false",
"channel_security_enabled": "true",
"channel_security_type": "OAuth",/*OAuth-Basic-Kerberos*/
"channel_exception_type": "log",/*log-Splunk*/
"channel_in_only": "false",
"channel_from_body": "",
"channel_from": "/isx/event/push",
"channel_from_type": "json",
"channel_from_transformer": "json",/*All Camel transformation type*/
"channel_from_transformer_metadata": {},
"channel_from_policies": null,/*Any business logic processor after consuming*/
"channel_from_method": "POST",
"channel_from_content_type": "application/json",
"channel_to_s": [
{
"channel_to": "{{isx-in}}?brokers={{broker-url}}&groupId={{topic-group}}&autoOffsetReset=earliest&topicIsPattern=true",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null, /*simple condition*/
"channel_to_function": null, /*Any Camel body function*/
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_kafka_fabric_wire",
"channel_to_type": "kafka:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null, /*Custom exception handler*/
"channel_to_policies": null,
"channel_to_client": [] /*Client type for connecting*/
},
{
"channel_to": "{{isx-in}}?exchangePattern=InOnly",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null,
"channel_to_function": null,
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_activemq_fabric_wire",
"channel_to_type": "activemq:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
},
{
"channel_to": "{{opensearch-provider}}/api/channel/opensearch/event/push/?bridgeEndpoint=true&indexName=isx-data-fabric&eventName=isx_event",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null,
"channel_to_function": null,
"channel_to_method": "POST",
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_opensearch",
"channel_to_type": "http:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
},
{ /* if successful and rest endpoint*/
"channel_to": "Event has been received",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_response_code": "${header.CamelHttpResponseCode}",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": "${channel.type}=='rest' && ${header.CamelHttpResponseCode}==200",
"channel_to_function": null,
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "build_send_successful_response",
"channel_to_type": "bean:",
"channel_to_transformer": "json",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
},
{ /* if failed and rest endpoint*/
"channel_to": "${header.channel.route.error}",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_response_code": "${header.CamelHttpResponseCode}",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": "${channel.type}=='rest' && ${header.CamelHttpResponseCode}!=200",
"channel_to_function": null,
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "build_send_failed_response",
"channel_to_type": "bean:",
"channel_to_transformer": "json",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
}
],
"channel_to_multicast": "true"
}MCF ACTIVEMQ XML Configuration Sample:
{
"channel_id": "mcf-channel-isx-data-xml",
"channel_src_id": "",
"channel_type": "from",
"channel_metric_enabled": "false",
"channel_security_enabled": "false",
"channel_security_type": null
"channel_exception_type": "log",
"channel_in_only": "false",
"channel_from_body": "",
"channel_from": "{{isx.in.queue}}",
"channel_from_type": "activemq:",
"channel_from_transformer": "jsonMXL",
"channel_from_transformer_metadata": {},
"channel_from_policies": null,
"channel_from_method": null,
"channel_from_content_type": "application/json",
"channel_to_s": [
{
"channel_to": "{{isx-in}}?brokers={{broker-url}}&groupId={{topic-group}}&autoOffsetReset=earliest&topicIsPattern=true",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null,
"channel_to_function": null,
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_kafka_fabric_wire",
"channel_to_type": "kafka:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": []
},
{
"channel_to": "{{isx-in}}?exchangePattern=InOnly",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null,
"channel_to_function": null,
"channel_to_method": null,
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_activemq_fabric_wire",
"channel_to_type": "activemq:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
},
{
"channel_to": "{{opensearch-provider}}/api/channel/opensearch/event/push/?bridgeEndpoint=true&indexName=isx-data-fabric&eventName=isx_event",
"channel_to_async": "false",
"channel_to_priority": "2",
"channel_to_retry": "0",
"channel_to_delay": "0",
"channel_to_dynamic": false,
"channel_to_retry_type": null,
"channel_to_standalone": "false",
"channel_to_content_condition": null,
"channel_to_function": null,
"channel_to_method": "POST",
"channel_to_body": "",
"channel_to_name": "push_isx_event_to_opensearch",
"channel_to_type": "http:",
"channel_to_transformer": "String",
"channel_to_transformer_metadata": {},
"channel_to_error_policies": null,
"channel_to_policies": null,
"channel_to_client": ""
},
],
"channel_to_multicast": "true"
}Note: The ActiveMQ XML configuration uses a channel_from_transformer of jsonXml to auto-transform between XML-to-JSON and back.
In summary, by embracing the power of MCF’s channel configuration approach, organizations can achieve efficient and flexible integration solutions without the need for intricate coding or manual configuration of Apache Camel routes. The streamlined integration development process, facilitated by MCF’s automation capabilities, empowers developers to focus on defining integration requirements, rapidly iterate on solutions, and effectively address dynamic business needs. With accelerated delivery and deployment, organizations can achieve faster time-to-market and gain a competitive edge in today’s dynamic business environment.
For more information on MCF and gaining access to our GitHub repository. We have streamlined our processes to ensure a seamless and efficient experience for our users. In order to provide you with the necessary information and access, we kindly request that you take a moment to complete the contact form below.
