Realtime Response using Talend ESB
10:31 AM
In ETL, there are scenarios when data needs to be processed as soon as client upload files into the system. Generally, we use triggers available in Data integration tools for such scenarios like wait for a file or we will trigger process in every 5 mins to check for the new file and process it. We always try to achieve realtime in loading data to our system.
Data integration tools do not provide much capabilities to respond to the client in realtime. It can provide fast and efficient way to process and validate data, but to give response to client in case of validation failure and not allowing them to upload wrong files by providing error messages in realtime can be achieved from service based system.
I have been facing same problem from 3 mins gap to reach realtime in loading data and to give response back to client in realtime if he commit any mistake in uploading.
I was using Talend Open Studio for Data Integration but to reach in realtime I shifted to Talend ESB v.6.1.1. Talend ESB studio has two prospective Integration and Mediation. Integration is same as from Data Integration studio and Talend ESB also comes with Runtime.
As both Data Integration studio and ESB studio support same components for data Integration I did not need to do any re-design for the logic in reading and writing data. Now in ESB I just need to wrap my ETL logic inside a RESTful web service which can be called from web browser with client specific parameters and can return response back in realtime.
Above snapshot has the request made to the dummy RESTful web service MyFirstAPI using Talend ESB.
In MyFirstAPI job I have used tRESTRequest to receive a request from browser using url pattern "talend/{action}" where action can be anything but job will return valid status only when action is valid. And using tFlowToIterate we can iterate request from clients and perform ETL specific to clients or actions.
I will try to cover how to create and deploy Talend ESB RESTful web service in Runtime environment in my future posts.
Please let me know your views I hope the above is useful to you.
As both Data Integration studio and ESB studio support same components for data Integration I did not need to do any re-design for the logic in reading and writing data. Now in ESB I just need to wrap my ETL logic inside a RESTful web service which can be called from web browser with client specific parameters and can return response back in realtime.
Above snapshot has the request made to the dummy RESTful web service MyFirstAPI using Talend ESB.
In MyFirstAPI job I have used tRESTRequest to receive a request from browser using url pattern "talend/{action}" where action can be anything but job will return valid status only when action is valid. And using tFlowToIterate we can iterate request from clients and perform ETL specific to clients or actions.
I will try to cover how to create and deploy Talend ESB RESTful web service in Runtime environment in my future posts.
Please let me know your views I hope the above is useful to you.

0 comments