ruby - Importing data that may take 10-15 minutes to process, what are my options in Rails? -

i have rails application displays thousands of products. products loaded product feeds, source may large xml file or web service api calls.

i want able re-use models in existing rails application in import process.

what options in importing data rails application?

1. i use sidekiq fire off rake tasks, not sure if sidekiq suitable tasks take 10+ minutes run? use cases have seen sending of emails , other similiar light tasks

2. i create maybe stand-alone ruby script, not sure how re-use rails models if go route.

update total product around 30-50k items.

sidekiq great option others have mentioned. 10+ minutes isn't unreasonable long understand if restart sidekiq process mid run job stopped well.

the concern have if importing 50k items , have failure near beginning you'll never last ones. suggest looking @ import routine , seeing if can break smaller components. this:

• start sidekiq import job.
• first thing job reschedule n hours later.
• fetch data api/xml.
• for each record in result schedule "import specific data" job data argument.
• done.

the key second last step. doing way primary job has better chance of succeeding doing reading api/xml , scheduling 50k more jobs. each of can run individually , if single 1 fails won't affect others.

the other thing remember unless configure not sidekiq rerun failed jobs. make sure "import specific data" job can run multiple times , still right thing.

i have similar setup has worked me 2 years.