FSCBatch
PHP 5.3 library to help you run huge batch.
It takes a PagerfantaAdapterInterface (doctrine orm, propel, array, solarium etc... available) as a data source, will get data in slice of the size of the batch size, and will then process each context in batch by calling the callback you provided.
<?php use FSCBatchBatch; use FSCBatchEventExecuteEvent; $usersAdapter = ...; $solrIndex = ...; $batch = new Batch($usersAdapter, function (ExecuteEvent $event) use ($solrIndexer) { $solrIndexer->indexUser($event->getContext()); }); $batch->run(10); // Execute in batch of 10
Features:
Evented system using the symfony2 event dispatcher, to easily be able to add behaviorsIncluded listeners:
ProgressEventListener: Displays progress, elapsed time and estimated remaining time at the end of each batch. DoctrineEventListener: at the end of each batch: flush() the object manager, to save everything at the same time (may improve performance in some cases) clear() the object manager, to avoid memory leaksExtra:
Add aPagerfantaAdapter
for doctrine ORM, that traverse the table using range queries on the id instead of LIMIT/OFFSET.
LIMIT/OFFSET degrades query time as the OFFSET grows, wheareas range queries time stay consistent.
Be aware that this library is a WIP, and requires more tests.
Examples
Simple batch
<?php require_once __DIR__.'/../vendor/autoload.php'; use FSCBatchBatch; use FSCBatchEventExecuteEvent; use FSCBatchEventListenerProgressEventListener; use PagerfantaAdapterArrayAdapter; use SymfonyComponentConsoleOutputConsoleOutput; $passwords = range(1, 100); $hashes = array(); $batch = new Batch(new ArrayAdapter($passwords), function (ExecuteEvent $event) use (&$hashes) { $hashes[] = crypt($event->getContext(), '$2a$10$'); }); $batch->getEventDispatcher()->addSubscriber(new ProgressEventListener(new ConsoleOutput())); $batch->run(10);
Would output
$ php examples/array_closure.php
Batch run start. 100 jobs [Mem: 0.52 MB]
[ 10/100] [ 10.00 %] ([Δ 0.83 sec] - [Elapsed 0.83 sec] - [Remaining 7 secs]) [Mem: 0.52 MB]
[ 20/100] [ 20.00 %] ([Δ 0.83 sec] - [Elapsed 1 sec] - [Remaining 6 secs]) [Mem: 0.52 MB]
[ 30/100] [ 30.00 %] ([Δ 0.83 sec] - [Elapsed 2 secs] - [Remaining 5 secs]) [Mem: 0.52 MB]
[ 40/100] [ 40.00 %] ([Δ 0.83 sec] - [Elapsed 3 secs] - [Remaining 4 secs]) [Mem: 0.52 MB]
[ 50/100] [ 50.00 %] ([Δ 0.82 sec] - [Elapsed 4 secs] - [Remaining 4 secs]) [Mem: 0.52 MB]
[ 60/100] [ 60.00 %] ([Δ 0.83 sec] - [Elapsed 4 secs] - [Remaining 3 secs]) [Mem: 0.52 MB]
[ 70/100] [ 70.00 %] ([Δ 0.83 sec] - [Elapsed 5 secs] - [Remaining 2 secs]) [Mem: 0.79 MB]
[ 80/100] [ 80.00 %] ([Δ 0.83 sec] - [Elapsed 6 secs] - [Remaining 1 sec]) [Mem: 0.79 MB]
[ 90/100] [ 90.00 %] ([Δ 0.83 sec] - [Elapsed 7 secs] - [Remaining 0.83 sec]) [Mem: 0.79 MB]
[100/100] [100.00 %] ([Δ 0.82 sec] - [Elapsed 8 secs] - [Remaining 0 sec]) [Mem: 0.79 MB]
Batch run end. took 0.82 sec [Mem: 0.79 MB]
Doctrine ORM Batch in a symfony command
This example uses the DoctrineEventListener, which flush (save everything) and clears the objectManager (avoid memory problems) at the end of each batch.
We also use a custom PagerfantaAdapter: DoctrineBatchAdapter
, that uses range queries (id > 100 AND id < 200) instead of LIMIT/OFFSET to avoid increasing query time as the OFFSET grows.
<?php use FSCBatchBatch; use FSCBatchCommandBatchCommand; use FSCBatchEventListenerDoctrineEventListener; use FSCBatchEventContextEvent; use PagerfantaAdapterDoctrineORMAdapter; class UserIndexSolrCommand extends BatchCommand { protected function createBatch() { $em = $this->getContainer()->get('doctrine')->getEntityManager(); $qb = $em->getRepository('User')->createQueryBuilder('u'); $batch = new Batch(new DoctrineORMAdapter($qb), array($this, 'indexUser')); $batch->getEventDispatcher()->addSubscriber(new DoctrineEventListener($em)); return $batch; } public function indexUser(ContextEvent $event) { $user = $event->getContext(); // Index this user! } }
版权声明:
1、该文章(资料)来源于互联网公开信息,我方只是对该内容做点评,所分享的下载地址为原作者公开地址。2、网站不提供资料下载,如需下载请到原作者页面进行下载。
3、本站所有内容均由合作方或网友上传,本站不对文档的完整性、权威性及其观点立场正确性做任何保证或承诺!文档内容仅供研究参考学习用!
4、如文档内容存在违规,或者侵犯商业秘密、侵犯著作权等,请点击“违规举报”。