azure - How to handle "The specified resource does not exist" exception while using Entity group transactions to purge WADLogs table -

azure - How to handle "The specified resource does not exist" exception while using Entity group transactions to purge WADLogs table -

we have requirement purge azure wadlogs table on periodic basis. achieving using entity grouping transactions delete records older 15 days. logic this.

bool recorddoesnotexistexceptionoccured = false; cloudtable wadlogstable = tableclient.gettablereference(wadlogstablename); partitionkey = "0" + datetime.utcnow.adddays(noofdays).ticks; tablequery<wadlogsentity> buildquery = new tablequery<wadlogsentity>().where( tablequery.generatefiltercondition("partitionkey", querycomparisons.lessthanorequal, partitionkey)); while (!recorddoesnotexistexceptionoccured) { ienumerable<wadlogsentity> result = wadlogstable.executequery(buildquery).take(1000); //// batch entity delete. if (result != null && result.count() > 0) { dictionary<string, tablebatchoperation> batches = new dictionary<string, tablebatchoperation>(); foreach (var entity in result) { tableoperation tableoperation = tableoperation.delete(entity); if (!batches.containskey(entity.partitionkey)) { batches.add(entity.partitionkey, new tablebatchoperation()); } // batch operation allows maximum 100 entities in batch must share same partitionkey. if (batches[entity.partitionkey].count < 100) { batches[entity.partitionkey].add(tableoperation); } } // execute batches. foreach (var batch in batches.values) { seek { await wadlogstable.executebatchasync(batch); } grab (exception exception) { // log exception here. // set flag. if (exception.message.contains(resourcedoesnotexist)) { recorddoesnotexistexceptionoccured = true; } break; } } } else { break; } }

my questions are:

is efficient way purge wadlogs table? if not, can create better? is right way handle "specified resource not exist exception"? if not, how can create better? would logic fail in particular case? how approach alter if code in worker has multiple instances deployed?

i have come code referencing solution given here.

batches in general best way purge part of table if can't delete whole thing. however, there couple of problems details in code above i'll outline below.

i'm not sure why you're getting error table entity returned in query unless you're deleting things table on different thread. however, scenario you've given should not halt whole process when error. had 1000 results delete. 2nd had been deleted thread , returned error when tried delete 1 time again - you're missing deleting of other entities, including other ones in batch since batches atomic! prepare scenario, i'd suggest removing failing entity batch (the error message include index), retrying batch, , continuing delete rest of batches.

see 2. additionally, if have more 100 entities same partition key not deleting remainder right away instead ignoring them , deleting them in next loop iteration. if there's many entries same partition key, result in lot of queries might more efficient add together them new batch.

once more workers going, issue in 2 worse current design since query first 1000. you'll have multiple threads downloading same results , trying delete same things. current design cause lot of errors since workers deleting same results. easiest solution see finding improve way split work doesn't cause overlap. 1 thought might have each worker take different day delete have added benefit of optimizing number of batches have sent. of course, there many other valid strategies.

azure windows-azure-storage azure-table-storage