feat: Reconcile Payments in background (#34596)
* feat: auto reconcile in background
* chore: Option to enable auto reconciliation in settings
* refactor: validate if feature is enabled in settings
* refactor: check for running job while using reconciliation tool
* chore: using doc to get filter values
* chore: use frappe.db.get_value in validations
* chore: cleanup commented out code
* chore: replace get_list with get_all
* chore: use block scope variable
* chore: type information for functions
* refactor: flag to ignore job validation check
* refactor: update parent doc status if all reconciled
* chore: create test_records file
* test: create a bunch of vouchers for testing auto reconcile
* chore: renamed auto_reconcile to process_payment_reconciliation
* chore: another child doctype to hold payments
* chore: remove duplicate field
* chore: add fetched payments to log
* chore: Popup comment message update
* chore: replace get_all with get_value
* chore: replace label in settings page
* chore: remove unit test and records
* refactor: status in reconciliation log
* refactor: set status in log as well
* chore: fix field name
* chore: change triggered job name
* chore: use status field in list view of log
* chore: status while there are no allocations
* refactor: split trigger function into two
* chore: adding cancelled status
* refactor: function trigger queued docs
* chore: cron job scheduled
* chore: fixing accouts settings json file
* chore: typos and variable scope
* chore: use 'pluck' in db call
* chore: remove redundant whitelist decorator
* chore: use single DB call to fetch values
* chore: replace get_all with get_value
* refactor: use raw db calls to fetch reconciliation log records
Using get_doc on `Process Payment Reconciliation Log` is costly when
handling large volumes of invoices.
Use raw frappe.db.get_all to selectively pull status and reconciled count
* chore: update status on successful batch operation
* chore: make payment table readonly
* chore: ability to pause the background job
* chore: remove isolate_each_allocation
* chore: more description in progress bar
* refactor: partially working state
* refactor: update reconcile flag and setting hard limits for fetching
* chore: make allocation editable -- NEED TO REVERT
* chore: pause button
* refactor: skip setter function in Payment Entry for better performan
* refactor: split reconcile function and skip a setter function
1. Split reconcile function into 2
2. While reconciling against payment entry, skip a
set_missing_ref_details setter method
* chore: increase payment limit
* refactor: replace frappe.db.get_all with frappe.db.get_value
* chore: remove unwanted doctypes
* refactor: make allocation table readonly
* perf: update ref_details only for newly linked invoices
* chore: rename skip flag
* refactor(UI): receivable_payable field should auto populate
* refactor: no control statements in finally block
* chore: cleanup section and rename checkbox
* chore: update new fieldname in code
* chore: update error msg
* refactor: start and pause integrated into status
pause checkbox has been removed
* refactor: added cancelled status to the log doctype
1. Moved the status section to the bottom in parent doc
2. Using alerts to indicate Job trigger status
(cherry picked from commit ed14d1ce44)
Co-authored-by: ruthra kumar <ruthra@erpnext.com>
1. No need to keep old PLE's on reconciliation.
2. Added Validation to catch debit-credit mismatch on JE's
3. Only update outstanding amount for newly reconciled invoices
(cherry picked from commit 11cf694d9a)
fix: fixes query builder order by issue in postgres (#32441)
* fix: fixes query builder order by issue in postgres
* fix: linter
(cherry picked from commit 69efd2ee24)
Co-authored-by: Shridhar Patil <shridhar.p@zerodha.com>
* refactor: DB independent quoting and truthy/falsy values
* style: reformat to black spec
* fix: ifnull -> coalesce
* fix: coalesce -> Coalesce
* fix: revert pypika comparison
* refactor: convert queries to QB
* fix: incorrect value types for query
`=` query makes no sense with list of values
* fix: remove warehouse docstatus condition
* fix: keep using base rate as rate
Co-authored-by: Ankush Menat <ankush@frappe.io>
If stock voucher count goes >1000 then fetching all gles and reposting
them all at once requires much more memory and can cause crash.
- This PR ensures that GLE reposting is done in chunks of 100 vouchers.
- This PR also starts keeping track of how many such chunks were
processed so in future progress is resumed in event of timeout.
If you have a huge list of docs to repost then maintaining transaction
throughtout entire GL reposting is not only unnecessary but also creates
performance issues. Periodically commiting the changes prevents lost
progress and reduces memory usage.
In Sales/Purchase invoices credit/debit are flipped and negated while making GLE,
this is unflipped while posting them but if we compare the flipped ones
it will always result in comparison failure and repost it.
- It runs after repost (after commiting), so it has 0 impact on repost itself
- Just that scheduled job log shows up as failed, while actually repost is done and repost document is "Completed"
- The check is inaccurate also, so best to remove this
- Users cant do anything about it anyway. This just piles up phantom failed jobs unneccessarily
