In apache airflow i am trying to impliment an etl

Question

In apache airflow i am trying to impliment an etl

process.

I wany to get the list of stores from the db, then for each store get sales data, fetch weather data from ext. api, apply forecasting function then load results into another table.

I can see two option, write a single script to do everything and execute as a single task. Split into different tasks, but then i would have to create tasks dynamicaly (for each store) and lot of data transfer. Which seems messy.

Any suggestions on wich approach is better?

0

22.02.2019

2 ответов

18 просмотров

Nithin Murali Автор вопроса

Also i think i can scale well with second approach

0

22.02.2019

Sharuzzaman Ahmat Raslan · Accepted Answer

Sharuzzaman Ahmat Raslan

Not sure about Apache Airflow, but if you use AWS, you can deploy this using AWS Step Function

0

22.02.2019

Похожие чаты

In apache airflow i am trying to impliment an etl

2 ответов

Похожие вопросы