Is it a constant stream of data or a fixed

Question

Go

David Cuadrado 🖍

Is it a constant stream of data or a fixed

set of data ?

0

11.04.2021

6 ответов

12 просмотров

jeffotoni

AYUB
There is a large csv file - First stage of the pip...

If you need to have an order if I understand correctly, if the order is the lines you are reading in your scan, using goroutines will not help you because the problem needs to be resolved synchronously 😔😔

0

11.04.2021

Alex

AYUB
There is a large csv file - First stage of the pip...

I'm not sure I have completely understood your problem but if the order is the line of the csv file you can pass to the go routine the line and the index. The go routine will return the result and the index. Who receive the result will put it in a map (index<->result). This is valid for both second and third stage.

0

12.04.2021

AYUB

jeffotoni
If you need to have an order if I understand corre...

Yeah that's what I came up with. I added some buffered channels for the pipeline stage communication so I can have a little bit of concurrency.

0

12.04.2021

jeffotoni

AYUB
Yeah that's what I came up with. I added some buff...

But if the whole process is synchronous, there is no need to use simultaneity. But if it is possible to use everything asynchronously, the conversation is different.

0

12.04.2021

AYUB

jeffotoni
But if the whole process is synchronous, there is ...

There might be some way to tranform the business logic into something more independent, to keep it async. For example, batching each set of records with the same ID. This way it could be more async

0

12.04.2021

AYUB · Accepted Answer

AYUB

There is a large csv file - First stage of the pipeline reads data line by line and pushes them into a channel - second stage of the pipeline reads data from the previous channel and filters invalid data, then pushes the valid data into another channel - third stage, reads data from the previous channel and calculates some stuff based on them, then writes the results into an output file

0

11.04.2021

Похожие чаты

Is it a constant stream of data or a fixed

6 ответов

Похожие вопросы