feat: improve locking: lock only after fetch completed

[faqiharifian]

What this PR does / why we need it:

When pubsub disabled and poll enabled, every time poller runs XP treatment service tries to acquire write lock and the treatment service refreshes the local storage by doing multiple calls to management service and update the data/maps.

This causes other part that use RLock to be blocked until all the calls and update are done.

This MR moved the write lock to after all the calls are completed, this reduces duration of RLock being blocked

[mbruner]

@faqiharifian I have one concern: right now the Init method does three things:

1. getAllProjects()
2. initProjectSegmenters()
3. initExperiments()
   All three steps are in one critical section.

With the proposed change, we split it into three independent critical sections, and it is possible, for example, new requests will be processed after initProjectSegmenters() and before initExperiments(). What is the chance that the state will be inconsistent?

What do you think about refactoring it to something like:

newSegmenters = s.initProjectSegmenters()
newExperiments = s.initExperiments()

s.Lock()
defer s.Unlock()
s.subscribedProjectSettings = ...
s.projectSegmenters = newSegmenters
s.Experiments = newExperiments

[bthari]

if the locking mechanism is not running in here yet, is it possible that there will be two client who requests the same ProjectID, and therefore the s.ProjectSettings in here will have duplicate item?

[faqiharifian]

I just moving things around to optimize the locking, there's no new logic added in this MR and this method is not used by the Poller changes we introduced previously. Will not do it in this MR.

[bthari]

shouldn't we put the !ContainsProjectId(s.subscribedProjectIds, projectId) before this function? It is possible that projectSettings above is still nil, not because just the project can't be find, but also we don't subscribe to this project. If this is intended with the new change, perhaps can modify the comment above this

[faqiharifian]

Not sure about the intention of this, I just moving things around to optimize the locking, including the comment