Hey guys. I'm working on a crawler that writes data

Question

Hey guys. I'm working on a crawler that writes data

to a file, crawlers can be multiple processes writing to the same file. So how shall I implement sync mutex on files between multiple go apps?

0

09.02.2021

10 ответов

7 просмотров

Thirteen

AYUB
Why don't you pass all outputs to a queue to be wr...

The guy never mentioned whether the processes are distributed i.e on diff machines, which will everything utterly impossible using file approach.

0

09.02.2021

AYUB

Thirteen
The guy never mentioned whether the processes are ...

Doesn't need a file. One binary, several goroutines crawling and pushing the result into a channel(queue). Then another goroutine(queue worker) consumes data to be written to the file one by one.

0

09.02.2021

Thirteen

AYUB
Doesn't need a file. One binary, several goroutine...

This is fine if scalability is not a key consideration.

0

09.02.2021

ㅤ Автор вопроса

AYUB
Why don't you pass all outputs to a queue to be wr...

What kind of a queue are we talking about? I thought of an html queue, that would require another API development

0

09.02.2021

ㅤ Автор вопроса

Thirteen
The guy never mentioned whether the processes are ...

It's on the same machine/server. Golang will crawl the web and put data in json files, PHP will process the data. I'm only responsible for Golang.

0

09.02.2021

ㅤ Автор вопроса

AYUB
Doesn't need a file. One binary, several goroutine...

Don't forget that I'm talking about multiple processes, if it was just one, I'd go with sync mutex or channels. What I'm looking for is a sync mutex shared between multiple apps/processes

0

09.02.2021

Mark X

ㅤ
What kind of a queue are we talking about? I thoug...

Shouldn’t be that much! You could keep it simple and write an endpoint which is accepting the data and it stupidly adds it to a channel. Then you can use your current file handler which is getting data from the channel

0

09.02.2021

Пользователь 61930

Mark X
Shouldn’t be that much! You could keep it simple a...

😷😴

0

09.02.2021

ㅤ Автор вопроса

Mark X
Shouldn’t be that much! You could keep it simple a...

It even made the bot that mentioned you to freak out lol. So I guess using a tcp channel is the closest I can get. Hope I could make them change their mind, or get some rest because I feel like I'm overthinking it

0

09.02.2021

AYUB · Accepted Answer

AYUB

Why don't you pass all outputs to a queue to be written in order? You could have a single queue worker reading from a chan or whatever, writing to the file. No race condition, everyone happy

0

09.02.2021

Похожие чаты

Hey guys. I'm working on a crawler that writes data

10 ответов

Похожие вопросы